From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.haj.ipfire.org (localhost [IPv6:::1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4djLgj47qZz332Y for ; Fri, 02 Jan 2026 11:14:41 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client CN "mail01.haj.ipfire.org", Issuer "R12" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4djLgf04NMz2xGm for ; Fri, 02 Jan 2026 11:14:38 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4djLgd31z7z2C; Fri, 02 Jan 2026 11:14:37 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1767352477; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/nrzWcfWnUN/Bk8/6XND6zTGsCt3F75fVcfA10Lnu3k=; b=RWkLGhsINyrSM0Jawbj5tTFxTRX+DqMzEWzO8fE72RZoSvX4X0afXmxGdT9VV2dJGFPyv2 rleEq7HFAoGKETAg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1767352477; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/nrzWcfWnUN/Bk8/6XND6zTGsCt3F75fVcfA10Lnu3k=; b=Tj0xgLlwwCjQonnvXwxu1zPXABou5h0fEOimQC5NpfXaG8G7sRAxxOghq3W0pRCtdDI19s sahj1LIFdUUrZHPc5rqNm0Fm2dBTTOPNoixsDulSHxgykSE8+xFXi/kySn0j25RoYAZ8Jy PDVm13wbvzJFkMf/RyDXhpgDDSigswDDAeKOAYrss67Mf+Z2wenM3raxG34XQjb5RZlSEB 7LH2874Juh12TvUboKnLu+NjlF8KiHCitKCXZyPT8dOpIthy2MCTQV+wAEP6TGzG2CvKuX 4Eomg6D/42F0sRksFyN6URL60lN53g5RU8j7Vmv+lHzIUbMLXa0MqKchThRPFw== Content-Type: text/plain; charset=utf-8 Precedence: list List-Id: List-Subscribe: , List-Unsubscribe: , List-Post: List-Help: Sender: Mail-Followup-To: Mime-Version: 1.0 Subject: Re: Let's launch our own blocklists... From: Michael Tremer In-Reply-To: Date: Fri, 2 Jan 2026 11:14:36 +0000 Cc: Adolf Belka , "IPFire: Development-List" Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Jon Murphy Hello Jon, I cannot see what is different in this email. It looks the same than the = previous one to me. -Michael > On 30 Dec 2025, at 15:52, Jon Murphy wrote: >=20 > Forgot this part=E2=80=A6 >=20 > Comments are below. >=20 >=20 > ------ Original Message ------ > =46rom "Adolf Belka" > To "Michael Tremer" > Cc "IPFire: Development-List" > Date 12/30/2025 8:05:20=E2=80=AFAM > Subject Re: Let's launch our own blocklists... >=20 >> Hi Michael, >>=20 >> On 29/12/2025 13:05, Michael Tremer wrote: >>> Hello everyone, >>>=20 >>> I hope everyone had a great Christmas and a couple of quiet days to = relax from all the stress that was the year 2025. >> Still relaxing. >>> Having a couple of quieter days, I have been working on a new, = little (hopefully) side project that has probably been high up on our = radar since the Shalla list has shut down in 2020, or maybe even = earlier. The goal of the project is to provide good lists with = categories of domain names which are usually used to block access to = these domains. >>>=20 >>> I simply call this IPFire DNSBL which is short for IPFire DNS = Blocklists. >>>=20 >>> How did we get here? >>>=20 >>> As stated before, the URL filter feature in IPFire has the problem = that there are not many good blocklists available any more. There used = to be a couple more - most famously the Shalla list - but we are now = down to a single list from the University of Toulouse. It is a great = list, but it is not always the best fit for all users. >>>=20 >>> Then there has been talk about whether we could implement more = blocking features into IPFire that don=E2=80=99t involve the proxy. Most = famously blocking over DNS. The problem here remains a the blocking = feature is only as good as the data that is fed into it. Some people = have been putting forward a number of lists that were suitable for them, = but they would not have replaced the blocking functionality as we know = it. Their aim is to provide =E2=80=9Cone list for everything=E2=80=9D = but that is not what people usually want. It is targeted at a classic = home user and the only separation that is being made is any = adult/porn/NSFW content which usually is put into a separate list. >>>=20 >>> It would have been technically possible to include these lists and = let the users decide, but that is not the aim of IPFire. We want to do = the job for the user so that their job is getting easier. Including = obscure lists that don=E2=80=99t have a clear outline of what they = actually want to block (=E2=80=9Cbad content=E2=80=9D is not a category) = and passing the burden of figuring out whether they need the = =E2=80=9CLight=E2=80=9D, =E2=80=9CNormal=E2=80=9D, =E2=80=9CPro=E2=80=9D, = =E2=80=9CPro++=E2=80=9D, =E2=80=9CUltimate=E2=80=9D or even a = =E2=80=9CVenti=E2=80=9D list with cream on top is really not going to = work. It is all confusing and will lead to a bad user experience. >>>=20 >>> An even bigger problem that is however completely impossible to = solve is bad licensing of these lists. A user has asked the publisher of = the HaGeZi list whether they could be included in IPFire and under what = terms. The response was that the list is available under the terms of = the GNU General Public License v3, but that does not seem to be true. = The list contains data from various sources. Many of them are licensed = under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, =E2=80=A6) and = unless there is a non-public agreement that this data may be = redistributed, there is a huge legal issue here. We would expose our = users to potential copyright infringement which we cannot do under any = circumstances. Furthermore many lists are available under a = non-commercial license which excludes them from being used in any kind = of business. Plenty of IPFire systems are running in businesses, if not = even the vast majority. >>>=20 >>> In short, these lists are completely unusable for us. Apart from = HaGeZi, I consider OISD to have the same problem. >>>=20 >>> Enough about all the things that are bad. Let=E2=80=99s talk about = the new, good things: >>>=20 >>> Many blacklists on the internet are an amalgamation of other lists. = These lists vary in quality with some of them being not that good and = without a clear focus and others being excellent data. Since we don=E2=80=99= t have the man power to start from scratch, I felt that we can copy the = concept that HaGeZi and OISD have started and simply create a new list = that is based on other lists at the beginning to have a good starting = point. That way, we have much better control over what is going on these = lists and we can shape and mould them as we need them. Most importantly, = we don=E2=80=99t create a single lists, but many lists that have a clear = focus and allow users to choose what they want to block and what not. >>>=20 >>> So the current experimental stage that I am in has these lists: >>>=20 >>> * Ads >>> * Dating >>> * DoH >>> * Gambling >>> * Malware >>> * Porn >>> * Social >>> * Violence >>>=20 >>> The categories have been determined by what source lists we have = available with good data and are compatible with our chosen license CC = BY-SA 4.0. This is the same license that we are using for the IPFire = Location database, too. >>>=20 >>> The main use-cases for any kind of blocking are to comply with legal = requirements in networks with children (i.e. schools) to remove any kind = of pornographic content, sometimes block social media as well. Gambling = and violence are commonly blocked, too. Even more common would be = filtering advertising and any malicious content. >>>=20 >>> The latter is especially difficult because so many source lists = throw phishing, spyware, malvertising, tracking and other things into = the same bucket. Here this is currently all in the malware list which = has therefore become quite large. I am not sure whether this will stay = like this in the future or if we will have to make some adjustments, but = that is exactly why this is now entering some larger testing. >>>=20 >>> What has been built so far? In order to put these lists together = properly, track any data about where it is coming from, I have built a = tool in Python available here: >>>=20 >>> https://git.ipfire.org/?p=3Ddnsbl.git;a=3Dsummary >>>=20 >>> This tool will automatically update all lists once an hour if there = have been any changes and export them in various formats. The exported = lists are available for download here: >>>=20 >>> https://dnsbl.ipfire.org/lists/ >> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the = custom url works fine. >>=20 >> However you need to remember not to put the https:// at the front of = the url otherwise the WUI page completes without any error messages but = leaves an error message in the system logs saying >>=20 >> URL filter blacklist - ERROR: Not a valid URL filter blacklist >>=20 >> I found this out the hard way. >>=20 >> The other thing I noticed is that if you already have the Toulouse = University list downloaded and you then change to the ipfire custom url = then all the existing Toulouse blocklists stay in the directory on = IPFire and so you end up with a huge number of category tick boxes, most = of which are the old Toulouse ones, which are still available to select = and it is not clear which ones are from Toulouse and which ones from = IPFire. >>=20 >> I think if the blocklist URL source is changed or a custom url is = provided the first step should be to remove the old ones already = existing. >> That might be a problem because users can also create their own = blocklists and I believe those go into the same directory. >>=20 >> Without clearing out the old blocklists you end up with a huge number = of checkboxes for lists but it is not clear what happens if there is a = category that has the same name for the Toulouse list and the IPFire = list such as gambling. I will have a look at that and see what happens. >>=20 >> Not sure what the best approach to this is. >=20 >=20 > To find the older files I used this: > find /var/ipfire/urlfilter/blacklists -mtime +365 -type f -ls >=20 > To delete the older files and folders I did this: > find /var/ipfire/urlfilter/blacklists -mtime +365 -type f -delete = -o -type d -empty -delete >=20 >=20 >>=20 >>=20 >> Manually deleting all contents of the urlfilter/blacklists/ directory = and then selecting the IPFire blocklist url for the custom url I end up = with only the 8 categories from the IPFire list. >>=20 >> I have tested some gambling sites from the IPFire list and the block = worked on some. On others the site no longer exists so there is nothing = to block or has been changed to an https site and in that case it went = straight through. Also if I chose the http version of the link, it was = automatically changed to https and went through without being blocked. >>=20 >>=20 >>> If you download and open any of the files, you will see a large = header that includes copyright information and lists all sources that = have been used to create the individual lists. This way we ensure = maximum transparency, comply with the terms of the individual licenses = of the source lists and give credit to the people who help us to put = together the most perfect list for our users. >>>=20 >>> I would like this to become a project that is not only being used in = IPFire. We can and will be compatible with other solutions like AdGuard, = PiHole so that people can use our lists if they would like to even = though they are not using IPFire. Hopefully, these users will also feed = back to us so that we can improve our lists over time and make them one = of the best options out there. >>>=20 >>> All lists are available as a simple text file that lists the = domains. Then there is a hosts file available as well as a DNS zone file = and an RPZ file. Each list is individually available to be used in = squidGuard and there is a larger tarball available with all lists that = can be used in IPFire=E2=80=99s URL Filter. >>=20 > I tested the URL Filter with the change to the autoupdate.urls. For = me it only picked up one URL but I think my Web Proxy is configured = wrong. >=20 > =E2=80=A2 Is the non-Transparent needed to utilize the IPFire DNSBL = (a.k.a. IPFire DNS Blocklists)?? >=20 > =E2=80=A2 Does Web Proxy Auto-Discovery Protocol (WPAD) need to be = setup? >=20 > I ask because I disabled the Web Proxy once Shalla Services stopped a = few years ago. >=20 > I think the answer is yes to get HTTPS sites recognized. >=20 >>=20 >>> I am planning to add Suricata/Snort signatures whenever I have time = to do so. Even though it is not a good idea to filter pornographic = content this way, I suppose that catching malware and blocking DoH are = good use-cases for an IPS. Time will tell=E2=80=A6 >>>=20 >>> As a start, we will make these lists available in IPFire=E2=80=99s = URL Filter and collect some feedback about how we are doing. Afterwards, = we can see where else we can take this project. >>>=20 >>> If you want to enable this on your system, simply add the URL to = your autoupdate.urls file like here: >>>=20 >>> = https://git.ipfire.org/?p=3Dpeople/ms/ipfire-2.x.git;a=3Dcommitdiff;h=3Dbf= 675bb937faa7617474b3cc84435af3b1f7f45f >> I also tested out adding the IPFire url to autoupdate.urls and that = also worked fine for me. >>=20 >> Regards, >> Adolf. >>> This email is just a brain dump from me to this list. I would be = happy to answer any questions about implementation details, etc. if = people are interested. Right now, this email is long enough already=E2=80=A6= >>>=20 >>> All the best, >>> -Michael >>=20 >> -- Sent from my laptop >>=20 >>=20 >>=20