Let's launch our own blocklists...

Let's launch our own blocklists...

[Michael Tremer]

Hello everyone,

I hope everyone had a great Christmas and a couple of quiet days to relax from all the stress that was the year 2025.

Having a couple of quieter days, I have been working on a new, little (hopefully) side project that has probably been high up on our radar since the Shalla list has shut down in 2020, or maybe even earlier. The goal of the project is to provide good lists with categories of domain names which are usually used to block access to these domains.

I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists.

How did we get here?

As stated before, the URL filter feature in IPFire has the problem that there are not many good blocklists available any more. There used to be a couple more - most famously the Shalla list - but we are now down to a single list from the University of Toulouse. It is a great list, but it is not always the best fit for all users.

Then there has been talk about whether we could implement more blocking features into IPFire that don’t involve the proxy. Most famously blocking over DNS. The problem here remains a the blocking feature is only as good as the data that is fed into it. Some people have been putting forward a number of lists that were suitable for them, but they would not have replaced the blocking functionality as we know it. Their aim is to provide “one list for everything” but that is not what people usually want. It is targeted at a classic home user and the only separation that is being made is any adult/porn/NSFW content which usually is put into a separate list.

It would have been technically possible to include these lists and let the users decide, but that is not the aim of IPFire. We want to do the job for the user so that their job is getting easier. Including obscure lists that don’t have a clear outline of what they actually want to block (“bad content” is not a category) and passing the burden of figuring out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with cream on top is really not going to work. It is all confusing and will lead to a bad user experience.

An even bigger problem that is however completely impossible to solve is bad licensing of these lists. A user has asked the publisher of the HaGeZi list whether they could be included in IPFire and under what terms. The response was that the list is available under the terms of the GNU General Public License v3, but that does not seem to be true. The list contains data from various sources. Many of them are licensed under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public agreement that this data may be redistributed, there is a huge legal issue here. We would expose our users to potential copyright infringement which we cannot do under any circumstances. Furthermore many lists are available under a non-commercial license which excludes them from being used in any kind of business. Plenty of IPFire systems are running in businesses, if not even the vast majority.

In short, these lists are completely unusable for us. Apart from HaGeZi, I consider OISD to have the same problem.

Enough about all the things that are bad. Let’s talk about the new, good things:

Many blacklists on the internet are an amalgamation of other lists. These lists vary in quality with some of them being not that good and without a clear focus and others being excellent data. Since we don’t have the man power to start from scratch, I felt that we can copy the concept that HaGeZi and OISD have started and simply create a new list that is based on other lists at the beginning to have a good starting point. That way, we have much better control over what is going on these lists and we can shape and mould them as we need them. Most importantly, we don’t create a single lists, but many lists that have a clear focus and allow users to choose what they want to block and what not.

So the current experimental stage that I am in has these lists:

  * Ads
  * Dating
  * DoH
  * Gambling
  * Malware
  * Porn
  * Social
  * Violence

The categories have been determined by what source lists we have available with good data and are compatible with our chosen license CC BY-SA 4.0. This is the same license that we are using for the IPFire Location database, too.

The main use-cases for any kind of blocking are to comply with legal requirements in networks with children (i.e. schools) to remove any kind of pornographic content, sometimes block social media as well. Gambling and violence are commonly blocked, too. Even more common would be filtering advertising and any malicious content.

The latter is especially difficult because so many source lists throw phishing, spyware, malvertising, tracking and other things into the same bucket. Here this is currently all in the malware list which has therefore become quite large. I am not sure whether this will stay like this in the future or if we will have to make some adjustments, but that is exactly why this is now entering some larger testing.

What has been built so far? In order to put these lists together properly, track any data about where it is coming from, I have built a tool in Python available here:

  https://git.ipfire.org/?p=dnsbl.git;a=summary

This tool will automatically update all lists once an hour if there have been any changes and export them in various formats. The exported lists are available for download here:

  https://dnsbl.ipfire.org/lists/

If you download and open any of the files, you will see a large header that includes copyright information and lists all sources that have been used to create the individual lists. This way we ensure maximum transparency, comply with the terms of the individual licenses of the source lists and give credit to the people who help us to put together the most perfect list for our users.

I would like this to become a project that is not only being used in IPFire. We can and will be compatible with other solutions like AdGuard, PiHole so that people can use our lists if they would like to even though they are not using IPFire. Hopefully, these users will also feed back to us so that we can improve our lists over time and make them one of the best options out there.

All lists are available as a simple text file that lists the domains. Then there is a hosts file available as well as a DNS zone file and an RPZ file. Each list is individually available to be used in squidGuard and there is a larger tarball available with all lists that can be used in IPFire’s URL Filter. I am planning to add Suricata/Snort signatures whenever I have time to do so. Even though it is not a good idea to filter pornographic content this way, I suppose that catching malware and blocking DoH are good use-cases for an IPS. Time will tell…

As a start, we will make these lists available in IPFire’s URL Filter and collect some feedback about how we are doing. Afterwards, we can see where else we can take this project.

If you want to enable this on your system, simply add the URL to your autoupdate.urls file like here:

  https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f

This email is just a brain dump from me to this list. I would be happy to answer any questions about implementation details, etc. if people are interested. Right now, this email is long enough already…

All the best,
-Michael

Re: Let's launch our own blocklists...

[Adolf Belka]

Hi Michael,

On 29/12/2025 13:05, Michael Tremer wrote:
> Hello everyone,
>
> I hope everyone had a great Christmas and a couple of quiet days to relax from all the stress that was the year 2025.
Still relaxing.
> Having a couple of quieter days, I have been working on a new, little (hopefully) side project that has probably been high up on our radar since the Shalla list has shut down in 2020, or maybe even earlier. The goal of the project is to provide good lists with categories of domain names which are usually used to block access to these domains.
>
> I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists.
>
> How did we get here?
>
> As stated before, the URL filter feature in IPFire has the problem that there are not many good blocklists available any more. There used to be a couple more - most famously the Shalla list - but we are now down to a single list from the University of Toulouse. It is a great list, but it is not always the best fit for all users.
>
> Then there has been talk about whether we could implement more blocking features into IPFire that don’t involve the proxy. Most famously blocking over DNS. The problem here remains a the blocking feature is only as good as the data that is fed into it. Some people have been putting forward a number of lists that were suitable for them, but they would not have replaced the blocking functionality as we know it. Their aim is to provide “one list for everything” but that is not what people usually want. It is targeted at a classic home user and the only separation that is being made is any adult/porn/NSFW content which usually is put into a separate list.
>
> It would have been technically possible to include these lists and let the users decide, but that is not the aim of IPFire. We want to do the job for the user so that their job is getting easier. Including obscure lists that don’t have a clear outline of what they actually want to block (“bad content” is not a category) and passing the burden of figuring out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with cream on top is really not going to work. It is all confusing and will lead to a bad user experience.
>
> An even bigger problem that is however completely impossible to solve is bad licensing of these lists. A user has asked the publisher of the HaGeZi list whether they could be included in IPFire and under what terms. The response was that the list is available under the terms of the GNU General Public License v3, but that does not seem to be true. The list contains data from various sources. Many of them are licensed under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public agreement that this data may be redistributed, there is a huge legal issue here. We would expose our users to potential copyright infringement which we cannot do under any circumstances. Furthermore many lists are available under a non-commercial license which excludes them from being used in any kind of business. Plenty of IPFire systems are running in businesses, if not even the vast majority.
>
> In short, these lists are completely unusable for us. Apart from HaGeZi, I consider OISD to have the same problem.
>
> Enough about all the things that are bad. Let’s talk about the new, good things:
>
> Many blacklists on the internet are an amalgamation of other lists. These lists vary in quality with some of them being not that good and without a clear focus and others being excellent data. Since we don’t have the man power to start from scratch, I felt that we can copy the concept that HaGeZi and OISD have started and simply create a new list that is based on other lists at the beginning to have a good starting point. That way, we have much better control over what is going on these lists and we can shape and mould them as we need them. Most importantly, we don’t create a single lists, but many lists that have a clear focus and allow users to choose what they want to block and what not.
>
> So the current experimental stage that I am in has these lists:
>
>    * Ads
>    * Dating
>    * DoH
>    * Gambling
>    * Malware
>    * Porn
>    * Social
>    * Violence
>
> The categories have been determined by what source lists we have available with good data and are compatible with our chosen license CC BY-SA 4.0. This is the same license that we are using for the IPFire Location database, too.
>
> The main use-cases for any kind of blocking are to comply with legal requirements in networks with children (i.e. schools) to remove any kind of pornographic content, sometimes block social media as well. Gambling and violence are commonly blocked, too. Even more common would be filtering advertising and any malicious content.
>
> The latter is especially difficult because so many source lists throw phishing, spyware, malvertising, tracking and other things into the same bucket. Here this is currently all in the malware list which has therefore become quite large. I am not sure whether this will stay like this in the future or if we will have to make some adjustments, but that is exactly why this is now entering some larger testing.
>
> What has been built so far? In order to put these lists together properly, track any data about where it is coming from, I have built a tool in Python available here:
>
>    https://git.ipfire.org/?p=dnsbl.git;a=summary
>
> This tool will automatically update all lists once an hour if there have been any changes and export them in various formats. The exported lists are available for download here:
>
>    https://dnsbl.ipfire.org/lists/
The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the 
custom url works fine.

However you need to remember not to put the https:// at the front of the 
url otherwise the WUI page completes without any error messages but 
leaves an error message in the system logs saying

URL filter blacklist - ERROR: Not a valid URL filter blacklist

I found this out the hard way.

The other thing I noticed is that if you already have the Toulouse 
University list downloaded and you then change to the ipfire custom url 
then all the existing Toulouse blocklists stay in the directory on 
IPFire and so you end up with a huge number of category tick boxes, most 
of which are the old Toulouse ones, which are still available to select 
and it is not clear which ones are from Toulouse and which ones from IPFire.

I think if the blocklist URL source is changed or a custom url is 
provided the first step should be to remove the old ones already existing.
That might be a problem because users can also create their own 
blocklists and I believe those go into the same directory.

Without clearing out the old blocklists you end up with a huge number of 
checkboxes for lists but it is not clear what happens if there is a 
category that has the same name for the Toulouse list and the IPFire 
list such as gambling. I will have a look at that and see what happens.

Not sure what the best approach to this is.

Manually deleting all contents of the urlfilter/blacklists/ directory 
and then selecting the IPFire blocklist url for the custom url I end up 
with only the 8 categories from the IPFire list.

I have tested some gambling sites from the IPFire list and the block 
worked on some. On others the site no longer exists so there is nothing 
to block or has been changed to an https site and in that case it went 
straight through. Also if I chose the http version of the link, it was 
automatically changed to https and went through without being blocked.


> If you download and open any of the files, you will see a large header that includes copyright information and lists all sources that have been used to create the individual lists. This way we ensure maximum transparency, comply with the terms of the individual licenses of the source lists and give credit to the people who help us to put together the most perfect list for our users.
>
> I would like this to become a project that is not only being used in IPFire. We can and will be compatible with other solutions like AdGuard, PiHole so that people can use our lists if they would like to even though they are not using IPFire. Hopefully, these users will also feed back to us so that we can improve our lists over time and make them one of the best options out there.
>
> All lists are available as a simple text file that lists the domains. Then there is a hosts file available as well as a DNS zone file and an RPZ file. Each list is individually available to be used in squidGuard and there is a larger tarball available with all lists that can be used in IPFire’s URL Filter. I am planning to add Suricata/Snort signatures whenever I have time to do so. Even though it is not a good idea to filter pornographic content this way, I suppose that catching malware and blocking DoH are good use-cases for an IPS. Time will tell…
>
> As a start, we will make these lists available in IPFire’s URL Filter and collect some feedback about how we are doing. Afterwards, we can see where else we can take this project.
>
> If you want to enable this on your system, simply add the URL to your autoupdate.urls file like here:
>
>    https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f
I also tested out adding the IPFire url to autoupdate.urls and that also 
worked fine for me.

Regards,
Adolf.
> This email is just a brain dump from me to this list. I would be happy to answer any questions about implementation details, etc. if people are interested. Right now, this email is long enough already…
>
> All the best,
> -Michael

-- 
Sent from my laptop

Re[2]: Let's launch our own blocklists...

[Jon Murphy]

------ Original Message ------
From "Adolf Belka" <adolf.belka@ipfire.org>
To "Michael Tremer" <michael.tremer@ipfire.org>
Cc "IPFire: Development-List" <development@lists.ipfire.org>
Date 12/30/2025 8:05:20 AM
Subject Re: Let's launch our own blocklists...

>Hi Michael,
>
>On 29/12/2025 13:05, Michael Tremer wrote:
>>Hello everyone,
>>
>>I hope everyone had a great Christmas and a couple of quiet days to relax from all the stress that was the year 2025.
>Still relaxing.
>>Having a couple of quieter days, I have been working on a new, little (hopefully) side project that has probably been high up on our radar since the Shalla list has shut down in 2020, or maybe even earlier. The goal of the project is to provide good lists with categories of domain names which are usually used to block access to these domains.
>>
>>I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists.
>>
>>How did we get here?
>>
>>As stated before, the URL filter feature in IPFire has the problem that there are not many good blocklists available any more. There used to be a couple more - most famously the Shalla list - but we are now down to a single list from the University of Toulouse. It is a great list, but it is not always the best fit for all users.
>>
>>Then there has been talk about whether we could implement more blocking features into IPFire that don’t involve the proxy. Most famously blocking over DNS. The problem here remains a the blocking feature is only as good as the data that is fed into it. Some people have been putting forward a number of lists that were suitable for them, but they would not have replaced the blocking functionality as we know it. Their aim is to provide “one list for everything” but that is not what people usually want. It is targeted at a classic home user and the only separation that is being made is any adult/porn/NSFW content which usually is put into a separate list.
>>
>>It would have been technically possible to include these lists and let the users decide, but that is not the aim of IPFire. We want to do the job for the user so that their job is getting easier. Including obscure lists that don’t have a clear outline of what they actually want to block (“bad content” is not a category) and passing the burden of figuring out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with cream on top is really not going to work. It is all confusing and will lead to a bad user experience.
>>
>>An even bigger problem that is however completely impossible to solve is bad licensing of these lists. A user has asked the publisher of the HaGeZi list whether they could be included in IPFire and under what terms. The response was that the list is available under the terms of the GNU General Public License v3, but that does not seem to be true. The list contains data from various sources. Many of them are licensed under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public agreement that this data may be redistributed, there is a huge legal issue here. We would expose our users to potential copyright infringement which we cannot do under any circumstances. Furthermore many lists are available under a non-commercial license which excludes them from being used in any kind of business. Plenty of IPFire systems are running in businesses, if not even the vast majority.
>>
>>In short, these lists are completely unusable for us. Apart from HaGeZi, I consider OISD to have the same problem.
>>
>>Enough about all the things that are bad. Let’s talk about the new, good things:
>>
>>Many blacklists on the internet are an amalgamation of other lists. These lists vary in quality with some of them being not that good and without a clear focus and others being excellent data. Since we don’t have the man power to start from scratch, I felt that we can copy the concept that HaGeZi and OISD have started and simply create a new list that is based on other lists at the beginning to have a good starting point. That way, we have much better control over what is going on these lists and we can shape and mould them as we need them. Most importantly, we don’t create a single lists, but many lists that have a clear focus and allow users to choose what they want to block and what not.
>>
>>So the current experimental stage that I am in has these lists:
>>
>>    * Ads
>>    * Dating
>>    * DoH
>>    * Gambling
>>    * Malware
>>    * Porn
>>    * Social
>>    * Violence
>>
>>The categories have been determined by what source lists we have available with good data and are compatible with our chosen license CC BY-SA 4.0. This is the same license that we are using for the IPFire Location database, too.
>>
>>The main use-cases for any kind of blocking are to comply with legal requirements in networks with children (i.e. schools) to remove any kind of pornographic content, sometimes block social media as well. Gambling and violence are commonly blocked, too. Even more common would be filtering advertising and any malicious content.
>>
>>The latter is especially difficult because so many source lists throw phishing, spyware, malvertising, tracking and other things into the same bucket. Here this is currently all in the malware list which has therefore become quite large. I am not sure whether this will stay like this in the future or if we will have to make some adjustments, but that is exactly why this is now entering some larger testing.
>>
>>What has been built so far? In order to put these lists together properly, track any data about where it is coming from, I have built a tool in Python available here:
>>
>>    https://git.ipfire.org/?p=dnsbl.git;a=summary
>>
>>This tool will automatically update all lists once an hour if there have been any changes and export them in various formats. The exported lists are available for download here:
>>
>>    https://dnsbl.ipfire.org/lists/
>The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the custom url works fine.
>
>However you need to remember not to put the https:// at the front of the url otherwise the WUI page completes without any error messages but leaves an error message in the system logs saying
>
>URL filter blacklist - ERROR: Not a valid URL filter blacklist
>
>I found this out the hard way.
>
>The other thing I noticed is that if you already have the Toulouse University list downloaded and you then change to the ipfire custom url then all the existing Toulouse blocklists stay in the directory on IPFire and so you end up with a huge number of category tick boxes, most of which are the old Toulouse ones, which are still available to select and it is not clear which ones are from Toulouse and which ones from IPFire.
>
>I think if the blocklist URL source is changed or a custom url is provided the first step should be to remove the old ones already existing.
>That might be a problem because users can also create their own blocklists and I believe those go into the same directory.
>
>Without clearing out the old blocklists you end up with a huge number of checkboxes for lists but it is not clear what happens if there is a category that has the same name for the Toulouse list and the IPFire list such as gambling. I will have a look at that and see what happens.
>
>Not sure what the best approach to this is.


To find the older files I used this:
     find /var/ipfire/urlfilter/blacklists -mtime +365 -type f -ls

To delete the older files and folders I did this:
     find /var/ipfire/urlfilter/blacklists -mtime +365 -type f -delete -o 
-type d -empty -delete


>
>
>Manually deleting all contents of the urlfilter/blacklists/ directory and then selecting the IPFire blocklist url for the custom url I end up with only the 8 categories from the IPFire list.
>
>I have tested some gambling sites from the IPFire list and the block worked on some. On others the site no longer exists so there is nothing to block or has been changed to an https site and in that case it went straight through. Also if I chose the http version of the link, it was automatically changed to https and went through without being blocked.
>
>
>>If you download and open any of the files, you will see a large header that includes copyright information and lists all sources that have been used to create the individual lists. This way we ensure maximum transparency, comply with the terms of the individual licenses of the source lists and give credit to the people who help us to put together the most perfect list for our users.
>>
>>I would like this to become a project that is not only being used in IPFire. We can and will be compatible with other solutions like AdGuard, PiHole so that people can use our lists if they would like to even though they are not using IPFire. Hopefully, these users will also feed back to us so that we can improve our lists over time and make them one of the best options out there.
>>
>>All lists are available as a simple text file that lists the domains. Then there is a hosts file available as well as a DNS zone file and an RPZ file. Each list is individually available to be used in squidGuard and there is a larger tarball available with all lists that can be used in IPFire’s URL Filter.
>
I tested the URL Filter with the change to the autoupdate.urls.  For me 
it only picked up one URL but I think my Web Proxy is configured wrong.

  • Is the non-Transparent needed to utilize the IPFire DNSBL (a.k.a. 
IPFire DNS Blocklists)??

  • Does Web Proxy Auto-Discovery Protocol (WPAD) need to be setup?

I ask because I disabled the Web Proxy once Shalla Services stopped a 
few years ago.

I think the answer is yes to get HTTPS sites recognized.

>
>>I am planning to add Suricata/Snort signatures whenever I have time to do so. Even though it is not a good idea to filter pornographic content this way, I suppose that catching malware and blocking DoH are good use-cases for an IPS. Time will tell…
>>
>>As a start, we will make these lists available in IPFire’s URL Filter and collect some feedback about how we are doing. Afterwards, we can see where else we can take this project.
>>
>>If you want to enable this on your system, simply add the URL to your autoupdate.urls file like here:
>>
>>    https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f
>I also tested out adding the IPFire url to autoupdate.urls and that also worked fine for me.
>
>Regards,
>Adolf.
>>This email is just a brain dump from me to this list. I would be happy to answer any questions about implementation details, etc. if people are interested. Right now, this email is long enough already…
>>
>>All the best,
>>-Michael
>
>-- Sent from my laptop
>
>
>

Re: Let's launch our own blocklists...

[Michael Tremer]

Hello,

> On 30 Dec 2025, at 14:05, Adolf Belka <adolf.belka@ipfire.org> wrote:
> 
> Hi Michael,
> 
> On 29/12/2025 13:05, Michael Tremer wrote:
>> Hello everyone,
>> 
>> I hope everyone had a great Christmas and a couple of quiet days to relax from all the stress that was the year 2025.
> Still relaxing.

Very good, so let’s have a strong start into 2026 now!

>> Having a couple of quieter days, I have been working on a new, little (hopefully) side project that has probably been high up on our radar since the Shalla list has shut down in 2020, or maybe even earlier. The goal of the project is to provide good lists with categories of domain names which are usually used to block access to these domains.
>> 
>> I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists.
>> 
>> How did we get here?
>> 
>> As stated before, the URL filter feature in IPFire has the problem that there are not many good blocklists available any more. There used to be a couple more - most famously the Shalla list - but we are now down to a single list from the University of Toulouse. It is a great list, but it is not always the best fit for all users.
>> 
>> Then there has been talk about whether we could implement more blocking features into IPFire that don’t involve the proxy. Most famously blocking over DNS. The problem here remains a the blocking feature is only as good as the data that is fed into it. Some people have been putting forward a number of lists that were suitable for them, but they would not have replaced the blocking functionality as we know it. Their aim is to provide “one list for everything” but that is not what people usually want. It is targeted at a classic home user and the only separation that is being made is any adult/porn/NSFW content which usually is put into a separate list.
>> 
>> It would have been technically possible to include these lists and let the users decide, but that is not the aim of IPFire. We want to do the job for the user so that their job is getting easier. Including obscure lists that don’t have a clear outline of what they actually want to block (“bad content” is not a category) and passing the burden of figuring out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with cream on top is really not going to work. It is all confusing and will lead to a bad user experience.
>> 
>> An even bigger problem that is however completely impossible to solve is bad licensing of these lists. A user has asked the publisher of the HaGeZi list whether they could be included in IPFire and under what terms. The response was that the list is available under the terms of the GNU General Public License v3, but that does not seem to be true. The list contains data from various sources. Many of them are licensed under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public agreement that this data may be redistributed, there is a huge legal issue here. We would expose our users to potential copyright infringement which we cannot do under any circumstances. Furthermore many lists are available under a non-commercial license which excludes them from being used in any kind of business. Plenty of IPFire systems are running in businesses, if not even the vast majority.
>> 
>> In short, these lists are completely unusable for us. Apart from HaGeZi, I consider OISD to have the same problem.
>> 
>> Enough about all the things that are bad. Let’s talk about the new, good things:
>> 
>> Many blacklists on the internet are an amalgamation of other lists. These lists vary in quality with some of them being not that good and without a clear focus and others being excellent data. Since we don’t have the man power to start from scratch, I felt that we can copy the concept that HaGeZi and OISD have started and simply create a new list that is based on other lists at the beginning to have a good starting point. That way, we have much better control over what is going on these lists and we can shape and mould them as we need them. Most importantly, we don’t create a single lists, but many lists that have a clear focus and allow users to choose what they want to block and what not.
>> 
>> So the current experimental stage that I am in has these lists:
>> 
>>   * Ads
>>   * Dating
>>   * DoH
>>   * Gambling
>>   * Malware
>>   * Porn
>>   * Social
>>   * Violence
>> 
>> The categories have been determined by what source lists we have available with good data and are compatible with our chosen license CC BY-SA 4.0. This is the same license that we are using for the IPFire Location database, too.
>> 
>> The main use-cases for any kind of blocking are to comply with legal requirements in networks with children (i.e. schools) to remove any kind of pornographic content, sometimes block social media as well. Gambling and violence are commonly blocked, too. Even more common would be filtering advertising and any malicious content.
>> 
>> The latter is especially difficult because so many source lists throw phishing, spyware, malvertising, tracking and other things into the same bucket. Here this is currently all in the malware list which has therefore become quite large. I am not sure whether this will stay like this in the future or if we will have to make some adjustments, but that is exactly why this is now entering some larger testing.
>> 
>> What has been built so far? In order to put these lists together properly, track any data about where it is coming from, I have built a tool in Python available here:
>> 
>>   https://git.ipfire.org/?p=dnsbl.git;a=summary
>> 
>> This tool will automatically update all lists once an hour if there have been any changes and export them in various formats. The exported lists are available for download here:
>> 
>>   https://dnsbl.ipfire.org/lists/
> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the custom url works fine.
> 
> However you need to remember not to put the https:// at the front of the url otherwise the WUI page completes without any error messages but leaves an error message in the system logs saying
> 
> URL filter blacklist - ERROR: Not a valid URL filter blacklist
> 
> I found this out the hard way.

Oh yes, I forgot that there is a field on the web UI. If that does not accept https:// as a prefix, please file a bug and we will fix it.

> The other thing I noticed is that if you already have the Toulouse University list downloaded and you then change to the ipfire custom url then all the existing Toulouse blocklists stay in the directory on IPFire and so you end up with a huge number of category tick boxes, most of which are the old Toulouse ones, which are still available to select and it is not clear which ones are from Toulouse and which ones from IPFire.

Yes, I got the same thing, too. I think this is a bug, too, because otherwise you would have a lot of unused categories lying around that will never be updated. You cannot even tell which ones are from the current list and which ones from the old list.

Long-term we could even consider to remove the Univ. Toulouse list entirely and only have our own lists available which would make the problem go away.

> I think if the blocklist URL source is changed or a custom url is provided the first step should be to remove the old ones already existing.
> That might be a problem because users can also create their own blocklists and I believe those go into the same directory.

Good thought. We of course cannot delete the custom lists.

> Without clearing out the old blocklists you end up with a huge number of checkboxes for lists but it is not clear what happens if there is a category that has the same name for the Toulouse list and the IPFire list such as gambling. I will have a look at that and see what happens.
> 
> Not sure what the best approach to this is.

I believe it is removing all old content.

> Manually deleting all contents of the urlfilter/blacklists/ directory and then selecting the IPFire blocklist url for the custom url I end up with only the 8 categories from the IPFire list.
> 
> I have tested some gambling sites from the IPFire list and the block worked on some. On others the site no longer exists so there is nothing to block or has been changed to an https site and in that case it went straight through. Also if I chose the http version of the link, it was automatically changed to https and went through without being blocked.

The entire IPFire infrastructure always requires HTTPS. If you start using HTTP, you will be automatically redirected. It is 2026 and we don’t need to talk HTTP any more :)

I am glad to hear that the list is actually blocking. It would have been bad if it didn’t. Now we have the big task to check out the “quality” - however that can be determined. I think this is what needs some time…

In the meantime I have set up a small page on our website:

  https://www.ipfire.org/dnsbl

I would like to run this as a first-class project inside IPFire like we are doing with IPFire Location. That means that we need to tell people about what we are doing. Hopefully this page is a little start.

Initially it has a couple of high-level bullet points about what we are trying to achieve. I don’t think the text is very good, yet, but it is the best I had in that moment. There is then also a list of the lists that we currently offer. For each list, a detailed page will tell you about the license, how many domains are listed, when the last update has been, the sources and even there is a history page that shows all the changes whenever they have happened.

Finally there is a section that explains “How To Use?” the list which I would love to extend to include AdGuard Plus and things like that as well as Pi-Hole and whatever else could use the list. In a later step we should go ahead and talk to any projects to include our list(s) into their dropdown so that people can enable them nice and easy.

Behind the web page there is an API service that is running on the host that is running the DNSBL. The frontend web app that is running www.ipfire.org <http://www.ipfire.org/> is connecting to that API service to fetch the current lists, any details and so on. That way, we can split the logic and avoid creating a huge monolith of a web app. This also means that page could be down a little as I am still working on the entire thing and will frequently restart it.

The API documentation is available here and the API is publicly available: https://api.dnsbl.ipfire.org/docs

The website/API allows to file reports for anything that does not seem to be right on any of the lists. I would like to keep it as an open process, however, long-term, this cannot cost us any time. In the current stage, the reports are getting filed and that is about it. I still need to build out some way for admins or moderators (I am not sure what kind of roles I want to have here) to accept or reject those reports.

In case of us receiving a domain from a source list, I would rather like to submit a report to upstream for them to de-list. That way, we don’t have any admin to do and we are contributing back to other list. That would be a very good thing to do. We cannot however throw tons of emails at some random upstream projects without co-ordinating this first. By not reporting upstream, we will probably over time create large whitelists and I am not sure if that is a good thing to do.

Finally, there is a search box that can be used to find out if a domain is listed on any of the lists.

>> If you download and open any of the files, you will see a large header that includes copyright information and lists all sources that have been used to create the individual lists. This way we ensure maximum transparency, comply with the terms of the individual licenses of the source lists and give credit to the people who help us to put together the most perfect list for our users.
>> 
>> I would like this to become a project that is not only being used in IPFire. We can and will be compatible with other solutions like AdGuard, PiHole so that people can use our lists if they would like to even though they are not using IPFire. Hopefully, these users will also feed back to us so that we can improve our lists over time and make them one of the best options out there.
>> 
>> All lists are available as a simple text file that lists the domains. Then there is a hosts file available as well as a DNS zone file and an RPZ file. Each list is individually available to be used in squidGuard and there is a larger tarball available with all lists that can be used in IPFire’s URL Filter. I am planning to add Suricata/Snort signatures whenever I have time to do so. Even though it is not a good idea to filter pornographic content this way, I suppose that catching malware and blocking DoH are good use-cases for an IPS. Time will tell…
>> 
>> As a start, we will make these lists available in IPFire’s URL Filter and collect some feedback about how we are doing. Afterwards, we can see where else we can take this project.
>> 
>> If you want to enable this on your system, simply add the URL to your autoupdate.urls file like here:
>> 
>>   https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f
> I also tested out adding the IPFire url to autoupdate.urls and that also worked fine for me.

Very good. Should we include this already with Core Update 200? I don’t think we would break anything, but we might already gain a couple more people who are helping us to test this all?

The next step would be to build and test our DNS infrastructure. In the “How To Use?” Section on the pages of the individual lists, you can already see some instructions on how to use the lists as an RPZ. In comparison to other “providers”, I would prefer if people would be using DNS to fetch the lists. This is simply to push out updates in a cheap way for us and also do it very regularly.

Initially, clients will pull the entire list using AXFR. There is no way around this as they need to have the data in the first place. After that, clients will only need the changes. As you can see in the history, the lists don’t actually change that often. Sometimes only once a day and therefore downloading the entire list again would be a huge waste of data, both on the client side, but also for us hosting then.

Some other providers update their lists “every 10 minutes”, and there won't be any changes whatsoever. We don’t do that. We will only export the lists again when they have actually changed. The timestamps on the files that we offer using HTTPS can be checked by clients so that they won’t re-download the list again if it has not been changed. But using HTTPS still means that we would have to re-download the entire list and not only the changes.

Using DNS and IXFR will update the lists by only transferring a few kilobytes and therefore we can have clients check once an hour if a list has actually changed and only send out the raw changes. That way, we will be able to serve millions of clients at very cheap cost and they will always have a very up to date list.

As far as I can see any DNS software that supports RPZs supports AXFR/IXFR with exception of Knot Resolver which expects the zone to be downloaded externally. There is a ticket for AXFR/IXFR support (https://gitlab.nic.cz/knot/knot-resolver/-/issues/195).

Initially, some of the lists have been *huge* which is why a simple HTTP download is not feasible. The porn list was over 100 MiB. We could have spent thousands on just traffic alone which I don’t have for this kind of project. It would also be unnecessary money being spent. There are simply better solutions out there. But then I built something that basically tests the data that we are receiving from upstream but simply checking if a listed domain still exists. The result was very astonishing to me.

So whenever someone adds a domain to the list, we will (eventually, but not immediately) check if we can resolve the domain’s SOA record. If not, we mark the domain as non-active and will no longer include them in the exported data. This brought down the porn list from just under 5 million domains to just 421k. On the sources page (https://www.ipfire.org/dnsbl/lists/porn/sources) I am listing the percentage of dead domains from each of them and the UT1 list has 94% dead domains. Wow.

If we cannot resolve the domain, neither can our users. So we would otherwise fill the lists with tons of domains that simply could never be reached. And if they cannot be reached, why would we block them? We would waste bandwidth and a lot of memory on each single client.

The other sources have similarly high rations of dead domains. Most of them are in the 50-80% range. Therefore I am happy that we are doing some extra work here to give our users much better data for their filtering.

So, if you like, please go and check out the RPZ blocking with Unbound. Instructions are on the page. I would be happy to hear how this is turning out.

Please let me know if there are any more questions, and I would be glad to answer them.

Happy New Year,
-Michael

> 
> Regards,
> Adolf.
>> This email is just a brain dump from me to this list. I would be happy to answer any questions about implementation details, etc. if people are interested. Right now, this email is long enough already…
>> 
>> All the best,
>> -Michael
> 
> -- 
> Sent from my laptop

Re: Let's launch our own blocklists...

[Michael Tremer]

Hello Jon,

> On 30 Dec 2025, at 15:49, Jon Murphy <jon.murphy@ipfire.org> wrote:
> 
> 
> 
> 
> ------ Original Message ------
> From "Adolf Belka" <adolf.belka@ipfire.org>
> To "Michael Tremer" <michael.tremer@ipfire.org>
> Cc "IPFire: Development-List" <development@lists.ipfire.org>
> Date 12/30/2025 8:05:20 AM
> Subject Re: Let's launch our own blocklists...
> 
>> Hi Michael,
>> 
>> On 29/12/2025 13:05, Michael Tremer wrote:
>>> Hello everyone,
>>> 
>>> I hope everyone had a great Christmas and a couple of quiet days to relax from all the stress that was the year 2025.
>> Still relaxing.
>>> Having a couple of quieter days, I have been working on a new, little (hopefully) side project that has probably been high up on our radar since the Shalla list has shut down in 2020, or maybe even earlier. The goal of the project is to provide good lists with categories of domain names which are usually used to block access to these domains.
>>> 
>>> I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists.
>>> 
>>> How did we get here?
>>> 
>>> As stated before, the URL filter feature in IPFire has the problem that there are not many good blocklists available any more. There used to be a couple more - most famously the Shalla list - but we are now down to a single list from the University of Toulouse. It is a great list, but it is not always the best fit for all users.
>>> 
>>> Then there has been talk about whether we could implement more blocking features into IPFire that don’t involve the proxy. Most famously blocking over DNS. The problem here remains a the blocking feature is only as good as the data that is fed into it. Some people have been putting forward a number of lists that were suitable for them, but they would not have replaced the blocking functionality as we know it. Their aim is to provide “one list for everything” but that is not what people usually want. It is targeted at a classic home user and the only separation that is being made is any adult/porn/NSFW content which usually is put into a separate list.
>>> 
>>> It would have been technically possible to include these lists and let the users decide, but that is not the aim of IPFire. We want to do the job for the user so that their job is getting easier. Including obscure lists that don’t have a clear outline of what they actually want to block (“bad content” is not a category) and passing the burden of figuring out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with cream on top is really not going to work. It is all confusing and will lead to a bad user experience.
>>> 
>>> An even bigger problem that is however completely impossible to solve is bad licensing of these lists. A user has asked the publisher of the HaGeZi list whether they could be included in IPFire and under what terms. The response was that the list is available under the terms of the GNU General Public License v3, but that does not seem to be true. The list contains data from various sources. Many of them are licensed under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public agreement that this data may be redistributed, there is a huge legal issue here. We would expose our users to potential copyright infringement which we cannot do under any circumstances. Furthermore many lists are available under a non-commercial license which excludes them from being used in any kind of business. Plenty of IPFire systems are running in businesses, if not even the vast majority.
>>> 
>>> In short, these lists are completely unusable for us. Apart from HaGeZi, I consider OISD to have the same problem.
>>> 
>>> Enough about all the things that are bad. Let’s talk about the new, good things:
>>> 
>>> Many blacklists on the internet are an amalgamation of other lists. These lists vary in quality with some of them being not that good and without a clear focus and others being excellent data. Since we don’t have the man power to start from scratch, I felt that we can copy the concept that HaGeZi and OISD have started and simply create a new list that is based on other lists at the beginning to have a good starting point. That way, we have much better control over what is going on these lists and we can shape and mould them as we need them. Most importantly, we don’t create a single lists, but many lists that have a clear focus and allow users to choose what they want to block and what not.
>>> 
>>> So the current experimental stage that I am in has these lists:
>>> 
>>>   * Ads
>>>   * Dating
>>>   * DoH
>>>   * Gambling
>>>   * Malware
>>>   * Porn
>>>   * Social
>>>   * Violence
>>> 
>>> The categories have been determined by what source lists we have available with good data and are compatible with our chosen license CC BY-SA 4.0. This is the same license that we are using for the IPFire Location database, too.
>>> 
>>> The main use-cases for any kind of blocking are to comply with legal requirements in networks with children (i.e. schools) to remove any kind of pornographic content, sometimes block social media as well. Gambling and violence are commonly blocked, too. Even more common would be filtering advertising and any malicious content.
>>> 
>>> The latter is especially difficult because so many source lists throw phishing, spyware, malvertising, tracking and other things into the same bucket. Here this is currently all in the malware list which has therefore become quite large. I am not sure whether this will stay like this in the future or if we will have to make some adjustments, but that is exactly why this is now entering some larger testing.
>>> 
>>> What has been built so far? In order to put these lists together properly, track any data about where it is coming from, I have built a tool in Python available here:
>>> 
>>>   https://git.ipfire.org/?p=dnsbl.git;a=summary
>>> 
>>> This tool will automatically update all lists once an hour if there have been any changes and export them in various formats. The exported lists are available for download here:
>>> 
>>>   https://dnsbl.ipfire.org/lists/
>> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the custom url works fine.
>> 
>> However you need to remember not to put the https:// at the front of the url otherwise the WUI page completes without any error messages but leaves an error message in the system logs saying
>> 
>> URL filter blacklist - ERROR: Not a valid URL filter blacklist
>> 
>> I found this out the hard way.
>> 
>> The other thing I noticed is that if you already have the Toulouse University list downloaded and you then change to the ipfire custom url then all the existing Toulouse blocklists stay in the directory on IPFire and so you end up with a huge number of category tick boxes, most of which are the old Toulouse ones, which are still available to select and it is not clear which ones are from Toulouse and which ones from IPFire.
>> 
>> I think if the blocklist URL source is changed or a custom url is provided the first step should be to remove the old ones already existing.
>> That might be a problem because users can also create their own blocklists and I believe those go into the same directory.
>> 
>> Without clearing out the old blocklists you end up with a huge number of checkboxes for lists but it is not clear what happens if there is a category that has the same name for the Toulouse list and the IPFire list such as gambling. I will have a look at that and see what happens.
>> 
>> Not sure what the best approach to this is.
> 
> 
> To find the older files I used this:
>    find /var/ipfire/urlfilter/blacklists -mtime +365 -type f -ls
> 
> To delete the older files and folders I did this:
>    find /var/ipfire/urlfilter/blacklists -mtime +365 -type f -delete -o -type d -empty -delete
> 
> 
>> 
>> 
>> Manually deleting all contents of the urlfilter/blacklists/ directory and then selecting the IPFire blocklist url for the custom url I end up with only the 8 categories from the IPFire list.
>> 
>> I have tested some gambling sites from the IPFire list and the block worked on some. On others the site no longer exists so there is nothing to block or has been changed to an https site and in that case it went straight through. Also if I chose the http version of the link, it was automatically changed to https and went through without being blocked.
>> 
>> 
>>> If you download and open any of the files, you will see a large header that includes copyright information and lists all sources that have been used to create the individual lists. This way we ensure maximum transparency, comply with the terms of the individual licenses of the source lists and give credit to the people who help us to put together the most perfect list for our users.
>>> 
>>> I would like this to become a project that is not only being used in IPFire. We can and will be compatible with other solutions like AdGuard, PiHole so that people can use our lists if they would like to even though they are not using IPFire. Hopefully, these users will also feed back to us so that we can improve our lists over time and make them one of the best options out there.
>>> 
>>> All lists are available as a simple text file that lists the domains. Then there is a hosts file available as well as a DNS zone file and an RPZ file. Each list is individually available to be used in squidGuard and there is a larger tarball available with all lists that can be used in IPFire’s URL Filter.
>> 
> I tested the URL Filter with the change to the autoupdate.urls.  For me it only picked up one URL but I think my Web Proxy is configured wrong.
> 
> • Is the non-Transparent needed to utilize the IPFire DNSBL (a.k.a. IPFire DNS Blocklists)??
> 
> • Does Web Proxy Auto-Discovery Protocol (WPAD) need to be setup?
> 
> I ask because I disabled the Web Proxy once Shalla Services stopped a few years ago.
> 
> I think the answer is yes to get HTTPS sites recognized.

You will have to set up the proxy explicitly in your web browser or use WPAD. If you only use transparent filtering, you will only catch any HTTP requests, but I suppose those virtually don’t exist any more. To filter HTTPS (which this list and the proxy do support), you will have to make the browser contact the proxy first and then try to contact the web page you want to download.

You should then see a lot of requests (potentially blocked) in the URL filter log. Make sure logging is enabled at the bottom of the page.

Moving forward I don’t think that the decline of web proxies and URL filter will be stopped by adding this list, however, there are still plenty of people using this and we would be able to give them much better data. The purpose of this list is of course not limited to the proxy and as the name suggests will probably be used somewhere else in the future.

-Michael

>> 
>>> I am planning to add Suricata/Snort signatures whenever I have time to do so. Even though it is not a good idea to filter pornographic content this way, I suppose that catching malware and blocking DoH are good use-cases for an IPS. Time will tell…
>>> 
>>> As a start, we will make these lists available in IPFire’s URL Filter and collect some feedback about how we are doing. Afterwards, we can see where else we can take this project.
>>> 
>>> If you want to enable this on your system, simply add the URL to your autoupdate.urls file like here:
>>> 
>>>   https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f
>> I also tested out adding the IPFire url to autoupdate.urls and that also worked fine for me.
>> 
>> Regards,
>> Adolf.
>>> This email is just a brain dump from me to this list. I would be happy to answer any questions about implementation details, etc. if people are interested. Right now, this email is long enough already…
>>> 
>>> All the best,
>>> -Michael
>> 
>> -- Sent from my laptop

Re: Let's launch our own blocklists...

[Adolf Belka]

Hi,

On 02/01/2026 12:09, Michael Tremer wrote:
> Hello,
> 
>> On 30 Dec 2025, at 14:05, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>
>> Hi Michael,
>>
>> On 29/12/2025 13:05, Michael Tremer wrote:
>>> Hello everyone,
>>>
>>> I hope everyone had a great Christmas and a couple of quiet days to relax from all the stress that was the year 2025.
>> Still relaxing.
> 
> Very good, so let’s have a strong start into 2026 now!

Starting next week, yes.

> 
>>> Having a couple of quieter days, I have been working on a new, little (hopefully) side project that has probably been high up on our radar since the Shalla list has shut down in 2020, or maybe even earlier. The goal of the project is to provide good lists with categories of domain names which are usually used to block access to these domains.
>>>
>>> I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists.
>>>
>>> How did we get here?
>>>
>>> As stated before, the URL filter feature in IPFire has the problem that there are not many good blocklists available any more. There used to be a couple more - most famously the Shalla list - but we are now down to a single list from the University of Toulouse. It is a great list, but it is not always the best fit for all users.
>>>
>>> Then there has been talk about whether we could implement more blocking features into IPFire that don’t involve the proxy. Most famously blocking over DNS. The problem here remains a the blocking feature is only as good as the data that is fed into it. Some people have been putting forward a number of lists that were suitable for them, but they would not have replaced the blocking functionality as we know it. Their aim is to provide “one list for everything” but that is not what people usually want. It is targeted at a classic home user and the only separation that is being made is any adult/porn/NSFW content which usually is put into a separate list.
>>>
>>> It would have been technically possible to include these lists and let the users decide, but that is not the aim of IPFire. We want to do the job for the user so that their job is getting easier. Including obscure lists that don’t have a clear outline of what they actually want to block (“bad content” is not a category) and passing the burden of figuring out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with cream on top is really not going to work. It is all confusing and will lead to a bad user experience.
>>>
>>> An even bigger problem that is however completely impossible to solve is bad licensing of these lists. A user has asked the publisher of the HaGeZi list whether they could be included in IPFire and under what terms. The response was that the list is available under the terms of the GNU General Public License v3, but that does not seem to be true. The list contains data from various sources. Many of them are licensed under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public agreement that this data may be redistributed, there is a huge legal issue here. We would expose our users to potential copyright infringement which we cannot do under any circumstances. Furthermore many lists are available under a non-commercial license which excludes them from being used in any kind of business. Plenty of IPFire systems are running in businesses, if not even the vast majority.
>>>
>>> In short, these lists are completely unusable for us. Apart from HaGeZi, I consider OISD to have the same problem.
>>>
>>> Enough about all the things that are bad. Let’s talk about the new, good things:
>>>
>>> Many blacklists on the internet are an amalgamation of other lists. These lists vary in quality with some of them being not that good and without a clear focus and others being excellent data. Since we don’t have the man power to start from scratch, I felt that we can copy the concept that HaGeZi and OISD have started and simply create a new list that is based on other lists at the beginning to have a good starting point. That way, we have much better control over what is going on these lists and we can shape and mould them as we need them. Most importantly, we don’t create a single lists, but many lists that have a clear focus and allow users to choose what they want to block and what not.
>>>
>>> So the current experimental stage that I am in has these lists:
>>>
>>>    * Ads
>>>    * Dating
>>>    * DoH
>>>    * Gambling
>>>    * Malware
>>>    * Porn
>>>    * Social
>>>    * Violence
>>>
>>> The categories have been determined by what source lists we have available with good data and are compatible with our chosen license CC BY-SA 4.0. This is the same license that we are using for the IPFire Location database, too.
>>>
>>> The main use-cases for any kind of blocking are to comply with legal requirements in networks with children (i.e. schools) to remove any kind of pornographic content, sometimes block social media as well. Gambling and violence are commonly blocked, too. Even more common would be filtering advertising and any malicious content.
>>>
>>> The latter is especially difficult because so many source lists throw phishing, spyware, malvertising, tracking and other things into the same bucket. Here this is currently all in the malware list which has therefore become quite large. I am not sure whether this will stay like this in the future or if we will have to make some adjustments, but that is exactly why this is now entering some larger testing.
>>>
>>> What has been built so far? In order to put these lists together properly, track any data about where it is coming from, I have built a tool in Python available here:
>>>
>>>    https://git.ipfire.org/?p=dnsbl.git;a=summary
>>>
>>> This tool will automatically update all lists once an hour if there have been any changes and export them in various formats. The exported lists are available for download here:
>>>
>>>    https://dnsbl.ipfire.org/lists/
>> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the custom url works fine.
>>
>> However you need to remember not to put the https:// at the front of the url otherwise the WUI page completes without any error messages but leaves an error message in the system logs saying
>>
>> URL filter blacklist - ERROR: Not a valid URL filter blacklist
>>
>> I found this out the hard way.
> 
> Oh yes, I forgot that there is a field on the web UI. If that does not accept https:// as a prefix, please file a bug and we will fix it.

I will confirm it and raise a bug.

> 
>> The other thing I noticed is that if you already have the Toulouse University list downloaded and you then change to the ipfire custom url then all the existing Toulouse blocklists stay in the directory on IPFire and so you end up with a huge number of category tick boxes, most of which are the old Toulouse ones, which are still available to select and it is not clear which ones are from Toulouse and which ones from IPFire.
> 
> Yes, I got the same thing, too. I think this is a bug, too, because otherwise you would have a lot of unused categories lying around that will never be updated. You cannot even tell which ones are from the current list and which ones from the old list.
> 
> Long-term we could even consider to remove the Univ. Toulouse list entirely and only have our own lists available which would make the problem go away.
> 
>> I think if the blocklist URL source is changed or a custom url is provided the first step should be to remove the old ones already existing.
>> That might be a problem because users can also create their own blocklists and I believe those go into the same directory.
> 
> Good thought. We of course cannot delete the custom lists.
> 
>> Without clearing out the old blocklists you end up with a huge number of checkboxes for lists but it is not clear what happens if there is a category that has the same name for the Toulouse list and the IPFire list such as gambling. I will have a look at that and see what happens.
>>
>> Not sure what the best approach to this is.
> 
> I believe it is removing all old content.
> 
>> Manually deleting all contents of the urlfilter/blacklists/ directory and then selecting the IPFire blocklist url for the custom url I end up with only the 8 categories from the IPFire list.
>>
>> I have tested some gambling sites from the IPFire list and the block worked on some. On others the site no longer exists so there is nothing to block or has been changed to an https site and in that case it went straight through. Also if I chose the http version of the link, it was automatically changed to https and went through without being blocked.
> 
> The entire IPFire infrastructure always requires HTTPS. If you start using HTTP, you will be automatically redirected. It is 2026 and we don’t need to talk HTTP any more :)

Some of the domains in the gambling list (maybe quite a lot) seem to 
only have an http access. If I tried https it came back with the fact 
that it couldn't find it.

> 
> I am glad to hear that the list is actually blocking. It would have been bad if it didn’t. Now we have the big task to check out the “quality” - however that can be determined. I think this is what needs some time…
> 
> In the meantime I have set up a small page on our website:
> 
>    https://www.ipfire.org/dnsbl
> 
> I would like to run this as a first-class project inside IPFire like we are doing with IPFire Location. That means that we need to tell people about what we are doing. Hopefully this page is a little start.
> 
> Initially it has a couple of high-level bullet points about what we are trying to achieve. I don’t think the text is very good, yet, but it is the best I had in that moment. There is then also a list of the lists that we currently offer. For each list, a detailed page will tell you about the license, how many domains are listed, when the last update has been, the sources and even there is a history page that shows all the changes whenever they have happened.
> 
> Finally there is a section that explains “How To Use?” the list which I would love to extend to include AdGuard Plus and things like that as well as Pi-Hole and whatever else could use the list. In a later step we should go ahead and talk to any projects to include our list(s) into their dropdown so that people can enable them nice and easy.
> 
> Behind the web page there is an API service that is running on the host that is running the DNSBL. The frontend web app that is running www.ipfire.org <http://www.ipfire.org/> is connecting to that API service to fetch the current lists, any details and so on. That way, we can split the logic and avoid creating a huge monolith of a web app. This also means that page could be down a little as I am still working on the entire thing and will frequently restart it.
> 
> The API documentation is available here and the API is publicly available: https://api.dnsbl.ipfire.org/docs
> 
> The website/API allows to file reports for anything that does not seem to be right on any of the lists. I would like to keep it as an open process, however, long-term, this cannot cost us any time. In the current stage, the reports are getting filed and that is about it. I still need to build out some way for admins or moderators (I am not sure what kind of roles I want to have here) to accept or reject those reports.
> 
> In case of us receiving a domain from a source list, I would rather like to submit a report to upstream for them to de-list. That way, we don’t have any admin to do and we are contributing back to other list. That would be a very good thing to do. We cannot however throw tons of emails at some random upstream projects without co-ordinating this first. By not reporting upstream, we will probably over time create large whitelists and I am not sure if that is a good thing to do.
> 
> Finally, there is a search box that can be used to find out if a domain is listed on any of the lists.
> 
>>> If you download and open any of the files, you will see a large header that includes copyright information and lists all sources that have been used to create the individual lists. This way we ensure maximum transparency, comply with the terms of the individual licenses of the source lists and give credit to the people who help us to put together the most perfect list for our users.
>>>
>>> I would like this to become a project that is not only being used in IPFire. We can and will be compatible with other solutions like AdGuard, PiHole so that people can use our lists if they would like to even though they are not using IPFire. Hopefully, these users will also feed back to us so that we can improve our lists over time and make them one of the best options out there.
>>>
>>> All lists are available as a simple text file that lists the domains. Then there is a hosts file available as well as a DNS zone file and an RPZ file. Each list is individually available to be used in squidGuard and there is a larger tarball available with all lists that can be used in IPFire’s URL Filter. I am planning to add Suricata/Snort signatures whenever I have time to do so. Even though it is not a good idea to filter pornographic content this way, I suppose that catching malware and blocking DoH are good use-cases for an IPS. Time will tell…
>>>
>>> As a start, we will make these lists available in IPFire’s URL Filter and collect some feedback about how we are doing. Afterwards, we can see where else we can take this project.
>>>
>>> If you want to enable this on your system, simply add the URL to your autoupdate.urls file like here:
>>>
>>>    https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f
>> I also tested out adding the IPFire url to autoupdate.urls and that also worked fine for me.
> 
> Very good. Should we include this already with Core Update 200? I don’t think we would break anything, but we might already gain a couple more people who are helping us to test this all?

I think that would be a good idea.

> 
> The next step would be to build and test our DNS infrastructure. In the “How To Use?” Section on the pages of the individual lists, you can already see some instructions on how to use the lists as an RPZ. In comparison to other “providers”, I would prefer if people would be using DNS to fetch the lists. This is simply to push out updates in a cheap way for us and also do it very regularly.
> 
> Initially, clients will pull the entire list using AXFR. There is no way around this as they need to have the data in the first place. After that, clients will only need the changes. As you can see in the history, the lists don’t actually change that often. Sometimes only once a day and therefore downloading the entire list again would be a huge waste of data, both on the client side, but also for us hosting then.
> 
> Some other providers update their lists “every 10 minutes”, and there won't be any changes whatsoever. We don’t do that. We will only export the lists again when they have actually changed. The timestamps on the files that we offer using HTTPS can be checked by clients so that they won’t re-download the list again if it has not been changed. But using HTTPS still means that we would have to re-download the entire list and not only the changes.
> 
> Using DNS and IXFR will update the lists by only transferring a few kilobytes and therefore we can have clients check once an hour if a list has actually changed and only send out the raw changes. That way, we will be able to serve millions of clients at very cheap cost and they will always have a very up to date list.
> 
> As far as I can see any DNS software that supports RPZs supports AXFR/IXFR with exception of Knot Resolver which expects the zone to be downloaded externally. There is a ticket for AXFR/IXFR support (https://gitlab.nic.cz/knot/knot-resolver/-/issues/195).
> 
> Initially, some of the lists have been *huge* which is why a simple HTTP download is not feasible. The porn list was over 100 MiB. We could have spent thousands on just traffic alone which I don’t have for this kind of project. It would also be unnecessary money being spent. There are simply better solutions out there. But then I built something that basically tests the data that we are receiving from upstream but simply checking if a listed domain still exists. The result was very astonishing to me.
> 
> So whenever someone adds a domain to the list, we will (eventually, but not immediately) check if we can resolve the domain’s SOA record. If not, we mark the domain as non-active and will no longer include them in the exported data. This brought down the porn list from just under 5 million domains to just 421k. On the sources page (https://www.ipfire.org/dnsbl/lists/porn/sources) I am listing the percentage of dead domains from each of them and the UT1 list has 94% dead domains. Wow.
> 
> If we cannot resolve the domain, neither can our users. So we would otherwise fill the lists with tons of domains that simply could never be reached. And if they cannot be reached, why would we block them? We would waste bandwidth and a lot of memory on each single client.
> 
> The other sources have similarly high rations of dead domains. Most of them are in the 50-80% range. Therefore I am happy that we are doing some extra work here to give our users much better data for their filtering.

Removing all dead entries sounds like an excellent step.

Regards,

Adolf.

> 
> So, if you like, please go and check out the RPZ blocking with Unbound. Instructions are on the page. I would be happy to hear how this is turning out.
> 
> Please let me know if there are any more questions, and I would be glad to answer them.
> 
> Happy New Year,
> -Michael
> 
>>
>> Regards,
>> Adolf.
>>> This email is just a brain dump from me to this list. I would be happy to answer any questions about implementation details, etc. if people are interested. Right now, this email is long enough already…
>>>
>>> All the best,
>>> -Michael
>>
>> -- 
>> Sent from my laptop
> 
> 
> 

-- 
Sent from my laptop

Re: Let's launch our own blocklists...

[Adolf Belka]

Hi Michael,

I have found that the malware list includes duckduckgo.com

Regards,
Adolf.


On 02/01/2026 14:02, Adolf Belka wrote:
> Hi,
>
> On 02/01/2026 12:09, Michael Tremer wrote:
>> Hello,
>>
>>> On 30 Dec 2025, at 14:05, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>
>>> Hi Michael,
>>>
>>> On 29/12/2025 13:05, Michael Tremer wrote:
>>>> Hello everyone,
>>>>
>>>> I hope everyone had a great Christmas and a couple of quiet days to 
>>>> relax from all the stress that was the year 2025.
>>> Still relaxing.
>>
>> Very good, so let’s have a strong start into 2026 now!
>
> Starting next week, yes.
>
>>
>>>> Having a couple of quieter days, I have been working on a new, 
>>>> little (hopefully) side project that has probably been high up on 
>>>> our radar since the Shalla list has shut down in 2020, or maybe 
>>>> even earlier. The goal of the project is to provide good lists with 
>>>> categories of domain names which are usually used to block access 
>>>> to these domains.
>>>>
>>>> I simply call this IPFire DNSBL which is short for IPFire DNS 
>>>> Blocklists.
>>>>
>>>> How did we get here?
>>>>
>>>> As stated before, the URL filter feature in IPFire has the problem 
>>>> that there are not many good blocklists available any more. There 
>>>> used to be a couple more - most famously the Shalla list - but we 
>>>> are now down to a single list from the University of Toulouse. It 
>>>> is a great list, but it is not always the best fit for all users.
>>>>
>>>> Then there has been talk about whether we could implement more 
>>>> blocking features into IPFire that don’t involve the proxy. Most 
>>>> famously blocking over DNS. The problem here remains a the blocking 
>>>> feature is only as good as the data that is fed into it. Some 
>>>> people have been putting forward a number of lists that were 
>>>> suitable for them, but they would not have replaced the blocking 
>>>> functionality as we know it. Their aim is to provide “one list for 
>>>> everything” but that is not what people usually want. It is 
>>>> targeted at a classic home user and the only separation that is 
>>>> being made is any adult/porn/NSFW content which usually is put into 
>>>> a separate list.
>>>>
>>>> It would have been technically possible to include these lists and 
>>>> let the users decide, but that is not the aim of IPFire. We want to 
>>>> do the job for the user so that their job is getting easier. 
>>>> Including obscure lists that don’t have a clear outline of what 
>>>> they actually want to block (“bad content” is not a category) and 
>>>> passing the burden of figuring out whether they need the “Light”, 
>>>> “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with 
>>>> cream on top is really not going to work. It is all confusing and 
>>>> will lead to a bad user experience.
>>>>
>>>> An even bigger problem that is however completely impossible to 
>>>> solve is bad licensing of these lists. A user has asked the 
>>>> publisher of the HaGeZi list whether they could be included in 
>>>> IPFire and under what terms. The response was that the list is 
>>>> available under the terms of the GNU General Public License v3, but 
>>>> that does not seem to be true. The list contains data from various 
>>>> sources. Many of them are licensed under incompatible licenses (CC 
>>>> BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public 
>>>> agreement that this data may be redistributed, there is a huge 
>>>> legal issue here. We would expose our users to potential copyright 
>>>> infringement which we cannot do under any circumstances. 
>>>> Furthermore many lists are available under a non-commercial license 
>>>> which excludes them from being used in any kind of business. Plenty 
>>>> of IPFire systems are running in businesses, if not even the vast 
>>>> majority.
>>>>
>>>> In short, these lists are completely unusable for us. Apart from 
>>>> HaGeZi, I consider OISD to have the same problem.
>>>>
>>>> Enough about all the things that are bad. Let’s talk about the new, 
>>>> good things:
>>>>
>>>> Many blacklists on the internet are an amalgamation of other lists. 
>>>> These lists vary in quality with some of them being not that good 
>>>> and without a clear focus and others being excellent data. Since we 
>>>> don’t have the man power to start from scratch, I felt that we can 
>>>> copy the concept that HaGeZi and OISD have started and simply 
>>>> create a new list that is based on other lists at the beginning to 
>>>> have a good starting point. That way, we have much better control 
>>>> over what is going on these lists and we can shape and mould them 
>>>> as we need them. Most importantly, we don’t create a single lists, 
>>>> but many lists that have a clear focus and allow users to choose 
>>>> what they want to block and what not.
>>>>
>>>> So the current experimental stage that I am in has these lists:
>>>>
>>>>    * Ads
>>>>    * Dating
>>>>    * DoH
>>>>    * Gambling
>>>>    * Malware
>>>>    * Porn
>>>>    * Social
>>>>    * Violence
>>>>
>>>> The categories have been determined by what source lists we have 
>>>> available with good data and are compatible with our chosen license 
>>>> CC BY-SA 4.0. This is the same license that we are using for the 
>>>> IPFire Location database, too.
>>>>
>>>> The main use-cases for any kind of blocking are to comply with 
>>>> legal requirements in networks with children (i.e. schools) to 
>>>> remove any kind of pornographic content, sometimes block social 
>>>> media as well. Gambling and violence are commonly blocked, too. 
>>>> Even more common would be filtering advertising and any malicious 
>>>> content.
>>>>
>>>> The latter is especially difficult because so many source lists 
>>>> throw phishing, spyware, malvertising, tracking and other things 
>>>> into the same bucket. Here this is currently all in the malware 
>>>> list which has therefore become quite large. I am not sure whether 
>>>> this will stay like this in the future or if we will have to make 
>>>> some adjustments, but that is exactly why this is now entering some 
>>>> larger testing.
>>>>
>>>> What has been built so far? In order to put these lists together 
>>>> properly, track any data about where it is coming from, I have 
>>>> built a tool in Python available here:
>>>>
>>>>    https://git.ipfire.org/?p=dnsbl.git;a=summary
>>>>
>>>> This tool will automatically update all lists once an hour if there 
>>>> have been any changes and export them in various formats. The 
>>>> exported lists are available for download here:
>>>>
>>>>    https://dnsbl.ipfire.org/lists/
>>> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the 
>>> custom url works fine.
>>>
>>> However you need to remember not to put the https:// at the front of 
>>> the url otherwise the WUI page completes without any error messages 
>>> but leaves an error message in the system logs saying
>>>
>>> URL filter blacklist - ERROR: Not a valid URL filter blacklist
>>>
>>> I found this out the hard way.
>>
>> Oh yes, I forgot that there is a field on the web UI. If that does 
>> not accept https:// as a prefix, please file a bug and we will fix it.
>
> I will confirm it and raise a bug.
>
>>
>>> The other thing I noticed is that if you already have the Toulouse 
>>> University list downloaded and you then change to the ipfire custom 
>>> url then all the existing Toulouse blocklists stay in the directory 
>>> on IPFire and so you end up with a huge number of category tick 
>>> boxes, most of which are the old Toulouse ones, which are still 
>>> available to select and it is not clear which ones are from Toulouse 
>>> and which ones from IPFire.
>>
>> Yes, I got the same thing, too. I think this is a bug, too, because 
>> otherwise you would have a lot of unused categories lying around that 
>> will never be updated. You cannot even tell which ones are from the 
>> current list and which ones from the old list.
>>
>> Long-term we could even consider to remove the Univ. Toulouse list 
>> entirely and only have our own lists available which would make the 
>> problem go away.
>>
>>> I think if the blocklist URL source is changed or a custom url is 
>>> provided the first step should be to remove the old ones already 
>>> existing.
>>> That might be a problem because users can also create their own 
>>> blocklists and I believe those go into the same directory.
>>
>> Good thought. We of course cannot delete the custom lists.
>>
>>> Without clearing out the old blocklists you end up with a huge 
>>> number of checkboxes for lists but it is not clear what happens if 
>>> there is a category that has the same name for the Toulouse list and 
>>> the IPFire list such as gambling. I will have a look at that and see 
>>> what happens.
>>>
>>> Not sure what the best approach to this is.
>>
>> I believe it is removing all old content.
>>
>>> Manually deleting all contents of the urlfilter/blacklists/ 
>>> directory and then selecting the IPFire blocklist url for the custom 
>>> url I end up with only the 8 categories from the IPFire list.
>>>
>>> I have tested some gambling sites from the IPFire list and the block 
>>> worked on some. On others the site no longer exists so there is 
>>> nothing to block or has been changed to an https site and in that 
>>> case it went straight through. Also if I chose the http version of 
>>> the link, it was automatically changed to https and went through 
>>> without being blocked.
>>
>> The entire IPFire infrastructure always requires HTTPS. If you start 
>> using HTTP, you will be automatically redirected. It is 2026 and we 
>> don’t need to talk HTTP any more :)
>
> Some of the domains in the gambling list (maybe quite a lot) seem to 
> only have an http access. If I tried https it came back with the fact 
> that it couldn't find it.
>
>>
>> I am glad to hear that the list is actually blocking. It would have 
>> been bad if it didn’t. Now we have the big task to check out the 
>> “quality” - however that can be determined. I think this is what 
>> needs some time…
>>
>> In the meantime I have set up a small page on our website:
>>
>>    https://www.ipfire.org/dnsbl
>>
>> I would like to run this as a first-class project inside IPFire like 
>> we are doing with IPFire Location. That means that we need to tell 
>> people about what we are doing. Hopefully this page is a little start.
>>
>> Initially it has a couple of high-level bullet points about what we 
>> are trying to achieve. I don’t think the text is very good, yet, but 
>> it is the best I had in that moment. There is then also a list of the 
>> lists that we currently offer. For each list, a detailed page will 
>> tell you about the license, how many domains are listed, when the 
>> last update has been, the sources and even there is a history page 
>> that shows all the changes whenever they have happened.
>>
>> Finally there is a section that explains “How To Use?” the list which 
>> I would love to extend to include AdGuard Plus and things like that 
>> as well as Pi-Hole and whatever else could use the list. In a later 
>> step we should go ahead and talk to any projects to include our 
>> list(s) into their dropdown so that people can enable them nice and 
>> easy.
>>
>> Behind the web page there is an API service that is running on the 
>> host that is running the DNSBL. The frontend web app that is running 
>> www.ipfire.org <http://www.ipfire.org/> is connecting to that API 
>> service to fetch the current lists, any details and so on. That way, 
>> we can split the logic and avoid creating a huge monolith of a web 
>> app. This also means that page could be down a little as I am still 
>> working on the entire thing and will frequently restart it.
>>
>> The API documentation is available here and the API is publicly 
>> available: https://api.dnsbl.ipfire.org/docs
>>
>> The website/API allows to file reports for anything that does not 
>> seem to be right on any of the lists. I would like to keep it as an 
>> open process, however, long-term, this cannot cost us any time. In 
>> the current stage, the reports are getting filed and that is about 
>> it. I still need to build out some way for admins or moderators (I am 
>> not sure what kind of roles I want to have here) to accept or reject 
>> those reports.
>>
>> In case of us receiving a domain from a source list, I would rather 
>> like to submit a report to upstream for them to de-list. That way, we 
>> don’t have any admin to do and we are contributing back to other 
>> list. That would be a very good thing to do. We cannot however throw 
>> tons of emails at some random upstream projects without co-ordinating 
>> this first. By not reporting upstream, we will probably over time 
>> create large whitelists and I am not sure if that is a good thing to do.
>>
>> Finally, there is a search box that can be used to find out if a 
>> domain is listed on any of the lists.
>>
>>>> If you download and open any of the files, you will see a large 
>>>> header that includes copyright information and lists all sources 
>>>> that have been used to create the individual lists. This way we 
>>>> ensure maximum transparency, comply with the terms of the 
>>>> individual licenses of the source lists and give credit to the 
>>>> people who help us to put together the most perfect list for our 
>>>> users.
>>>>
>>>> I would like this to become a project that is not only being used 
>>>> in IPFire. We can and will be compatible with other solutions like 
>>>> AdGuard, PiHole so that people can use our lists if they would like 
>>>> to even though they are not using IPFire. Hopefully, these users 
>>>> will also feed back to us so that we can improve our lists over 
>>>> time and make them one of the best options out there.
>>>>
>>>> All lists are available as a simple text file that lists the 
>>>> domains. Then there is a hosts file available as well as a DNS zone 
>>>> file and an RPZ file. Each list is individually available to be 
>>>> used in squidGuard and there is a larger tarball available with all 
>>>> lists that can be used in IPFire’s URL Filter. I am planning to add 
>>>> Suricata/Snort signatures whenever I have time to do so. Even 
>>>> though it is not a good idea to filter pornographic content this 
>>>> way, I suppose that catching malware and blocking DoH are good 
>>>> use-cases for an IPS. Time will tell…
>>>>
>>>> As a start, we will make these lists available in IPFire’s URL 
>>>> Filter and collect some feedback about how we are doing. 
>>>> Afterwards, we can see where else we can take this project.
>>>>
>>>> If you want to enable this on your system, simply add the URL to 
>>>> your autoupdate.urls file like here:
>>>>
>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f
>>> I also tested out adding the IPFire url to autoupdate.urls and that 
>>> also worked fine for me.
>>
>> Very good. Should we include this already with Core Update 200? I 
>> don’t think we would break anything, but we might already gain a 
>> couple more people who are helping us to test this all?
>
> I think that would be a good idea.
>
>>
>> The next step would be to build and test our DNS infrastructure. In 
>> the “How To Use?” Section on the pages of the individual lists, you 
>> can already see some instructions on how to use the lists as an RPZ. 
>> In comparison to other “providers”, I would prefer if people would be 
>> using DNS to fetch the lists. This is simply to push out updates in a 
>> cheap way for us and also do it very regularly.
>>
>> Initially, clients will pull the entire list using AXFR. There is no 
>> way around this as they need to have the data in the first place. 
>> After that, clients will only need the changes. As you can see in the 
>> history, the lists don’t actually change that often. Sometimes only 
>> once a day and therefore downloading the entire list again would be a 
>> huge waste of data, both on the client side, but also for us hosting 
>> then.
>>
>> Some other providers update their lists “every 10 minutes”, and there 
>> won't be any changes whatsoever. We don’t do that. We will only 
>> export the lists again when they have actually changed. The 
>> timestamps on the files that we offer using HTTPS can be checked by 
>> clients so that they won’t re-download the list again if it has not 
>> been changed. But using HTTPS still means that we would have to 
>> re-download the entire list and not only the changes.
>>
>> Using DNS and IXFR will update the lists by only transferring a few 
>> kilobytes and therefore we can have clients check once an hour if a 
>> list has actually changed and only send out the raw changes. That 
>> way, we will be able to serve millions of clients at very cheap cost 
>> and they will always have a very up to date list.
>>
>> As far as I can see any DNS software that supports RPZs supports 
>> AXFR/IXFR with exception of Knot Resolver which expects the zone to 
>> be downloaded externally. There is a ticket for AXFR/IXFR support 
>> (https://gitlab.nic.cz/knot/knot-resolver/-/issues/195).
>>
>> Initially, some of the lists have been *huge* which is why a simple 
>> HTTP download is not feasible. The porn list was over 100 MiB. We 
>> could have spent thousands on just traffic alone which I don’t have 
>> for this kind of project. It would also be unnecessary money being 
>> spent. There are simply better solutions out there. But then I built 
>> something that basically tests the data that we are receiving from 
>> upstream but simply checking if a listed domain still exists. The 
>> result was very astonishing to me.
>>
>> So whenever someone adds a domain to the list, we will (eventually, 
>> but not immediately) check if we can resolve the domain’s SOA record. 
>> If not, we mark the domain as non-active and will no longer include 
>> them in the exported data. This brought down the porn list from just 
>> under 5 million domains to just 421k. On the sources page 
>> (https://www.ipfire.org/dnsbl/lists/porn/sources) I am listing the 
>> percentage of dead domains from each of them and the UT1 list has 94% 
>> dead domains. Wow.
>>
>> If we cannot resolve the domain, neither can our users. So we would 
>> otherwise fill the lists with tons of domains that simply could never 
>> be reached. And if they cannot be reached, why would we block them? 
>> We would waste bandwidth and a lot of memory on each single client.
>>
>> The other sources have similarly high rations of dead domains. Most 
>> of them are in the 50-80% range. Therefore I am happy that we are 
>> doing some extra work here to give our users much better data for 
>> their filtering.
>
> Removing all dead entries sounds like an excellent step.
>
> Regards,
>
> Adolf.
>
>>
>> So, if you like, please go and check out the RPZ blocking with 
>> Unbound. Instructions are on the page. I would be happy to hear how 
>> this is turning out.
>>
>> Please let me know if there are any more questions, and I would be 
>> glad to answer them.
>>
>> Happy New Year,
>> -Michael
>>
>>>
>>> Regards,
>>> Adolf.
>>>> This email is just a brain dump from me to this list. I would be 
>>>> happy to answer any questions about implementation details, etc. if 
>>>> people are interested. Right now, this email is long enough already…
>>>>
>>>> All the best,
>>>> -Michael
>>>
>>> -- 
>>> Sent from my laptop
>>
>>
>>
>

-- 
Sent from my laptop

Re: Let's launch our own blocklists...

[Adolf Belka]

Hi Michael,


On 05/01/2026 12:11, Adolf Belka wrote:
> Hi Michael,
>
> I have found that the malware list includes duckduckgo.com
>
I have checked through the various sources used for the malware list.

The ShadowWhisperer (Tracking) list has improving.duckduckgo.com in its 
list. I suspect that this one is the one causing the problem.

The mtxadmin (_malware_typo) list has duckduckgo.com mentioned 3 times 
but not directly as a domain name - looks more like a reference.

Regards,

Adolf.


> Regards,
> Adolf.
>
>
> On 02/01/2026 14:02, Adolf Belka wrote:
>> Hi,
>>
>> On 02/01/2026 12:09, Michael Tremer wrote:
>>> Hello,
>>>
>>>> On 30 Dec 2025, at 14:05, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>
>>>> Hi Michael,
>>>>
>>>> On 29/12/2025 13:05, Michael Tremer wrote:
>>>>> Hello everyone,
>>>>>
>>>>> I hope everyone had a great Christmas and a couple of quiet days 
>>>>> to relax from all the stress that was the year 2025.
>>>> Still relaxing.
>>>
>>> Very good, so let’s have a strong start into 2026 now!
>>
>> Starting next week, yes.
>>
>>>
>>>>> Having a couple of quieter days, I have been working on a new, 
>>>>> little (hopefully) side project that has probably been high up on 
>>>>> our radar since the Shalla list has shut down in 2020, or maybe 
>>>>> even earlier. The goal of the project is to provide good lists 
>>>>> with categories of domain names which are usually used to block 
>>>>> access to these domains.
>>>>>
>>>>> I simply call this IPFire DNSBL which is short for IPFire DNS 
>>>>> Blocklists.
>>>>>
>>>>> How did we get here?
>>>>>
>>>>> As stated before, the URL filter feature in IPFire has the problem 
>>>>> that there are not many good blocklists available any more. There 
>>>>> used to be a couple more - most famously the Shalla list - but we 
>>>>> are now down to a single list from the University of Toulouse. It 
>>>>> is a great list, but it is not always the best fit for all users.
>>>>>
>>>>> Then there has been talk about whether we could implement more 
>>>>> blocking features into IPFire that don’t involve the proxy. Most 
>>>>> famously blocking over DNS. The problem here remains a the 
>>>>> blocking feature is only as good as the data that is fed into it. 
>>>>> Some people have been putting forward a number of lists that were 
>>>>> suitable for them, but they would not have replaced the blocking 
>>>>> functionality as we know it. Their aim is to provide “one list for 
>>>>> everything” but that is not what people usually want. It is 
>>>>> targeted at a classic home user and the only separation that is 
>>>>> being made is any adult/porn/NSFW content which usually is put 
>>>>> into a separate list.
>>>>>
>>>>> It would have been technically possible to include these lists and 
>>>>> let the users decide, but that is not the aim of IPFire. We want 
>>>>> to do the job for the user so that their job is getting easier. 
>>>>> Including obscure lists that don’t have a clear outline of what 
>>>>> they actually want to block (“bad content” is not a category) and 
>>>>> passing the burden of figuring out whether they need the “Light”, 
>>>>> “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with 
>>>>> cream on top is really not going to work. It is all confusing and 
>>>>> will lead to a bad user experience.
>>>>>
>>>>> An even bigger problem that is however completely impossible to 
>>>>> solve is bad licensing of these lists. A user has asked the 
>>>>> publisher of the HaGeZi list whether they could be included in 
>>>>> IPFire and under what terms. The response was that the list is 
>>>>> available under the terms of the GNU General Public License v3, 
>>>>> but that does not seem to be true. The list contains data from 
>>>>> various sources. Many of them are licensed under incompatible 
>>>>> licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a 
>>>>> non-public agreement that this data may be redistributed, there is 
>>>>> a huge legal issue here. We would expose our users to potential 
>>>>> copyright infringement which we cannot do under any circumstances. 
>>>>> Furthermore many lists are available under a non-commercial 
>>>>> license which excludes them from being used in any kind of 
>>>>> business. Plenty of IPFire systems are running in businesses, if 
>>>>> not even the vast majority.
>>>>>
>>>>> In short, these lists are completely unusable for us. Apart from 
>>>>> HaGeZi, I consider OISD to have the same problem.
>>>>>
>>>>> Enough about all the things that are bad. Let’s talk about the 
>>>>> new, good things:
>>>>>
>>>>> Many blacklists on the internet are an amalgamation of other 
>>>>> lists. These lists vary in quality with some of them being not 
>>>>> that good and without a clear focus and others being excellent 
>>>>> data. Since we don’t have the man power to start from scratch, I 
>>>>> felt that we can copy the concept that HaGeZi and OISD have 
>>>>> started and simply create a new list that is based on other lists 
>>>>> at the beginning to have a good starting point. That way, we have 
>>>>> much better control over what is going on these lists and we can 
>>>>> shape and mould them as we need them. Most importantly, we don’t 
>>>>> create a single lists, but many lists that have a clear focus and 
>>>>> allow users to choose what they want to block and what not.
>>>>>
>>>>> So the current experimental stage that I am in has these lists:
>>>>>
>>>>>    * Ads
>>>>>    * Dating
>>>>>    * DoH
>>>>>    * Gambling
>>>>>    * Malware
>>>>>    * Porn
>>>>>    * Social
>>>>>    * Violence
>>>>>
>>>>> The categories have been determined by what source lists we have 
>>>>> available with good data and are compatible with our chosen 
>>>>> license CC BY-SA 4.0. This is the same license that we are using 
>>>>> for the IPFire Location database, too.
>>>>>
>>>>> The main use-cases for any kind of blocking are to comply with 
>>>>> legal requirements in networks with children (i.e. schools) to 
>>>>> remove any kind of pornographic content, sometimes block social 
>>>>> media as well. Gambling and violence are commonly blocked, too. 
>>>>> Even more common would be filtering advertising and any malicious 
>>>>> content.
>>>>>
>>>>> The latter is especially difficult because so many source lists 
>>>>> throw phishing, spyware, malvertising, tracking and other things 
>>>>> into the same bucket. Here this is currently all in the malware 
>>>>> list which has therefore become quite large. I am not sure whether 
>>>>> this will stay like this in the future or if we will have to make 
>>>>> some adjustments, but that is exactly why this is now entering 
>>>>> some larger testing.
>>>>>
>>>>> What has been built so far? In order to put these lists together 
>>>>> properly, track any data about where it is coming from, I have 
>>>>> built a tool in Python available here:
>>>>>
>>>>>    https://git.ipfire.org/?p=dnsbl.git;a=summary
>>>>>
>>>>> This tool will automatically update all lists once an hour if 
>>>>> there have been any changes and export them in various formats. 
>>>>> The exported lists are available for download here:
>>>>>
>>>>>    https://dnsbl.ipfire.org/lists/
>>>> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the 
>>>> custom url works fine.
>>>>
>>>> However you need to remember not to put the https:// at the front 
>>>> of the url otherwise the WUI page completes without any error 
>>>> messages but leaves an error message in the system logs saying
>>>>
>>>> URL filter blacklist - ERROR: Not a valid URL filter blacklist
>>>>
>>>> I found this out the hard way.
>>>
>>> Oh yes, I forgot that there is a field on the web UI. If that does 
>>> not accept https:// as a prefix, please file a bug and we will fix it.
>>
>> I will confirm it and raise a bug.
>>
>>>
>>>> The other thing I noticed is that if you already have the Toulouse 
>>>> University list downloaded and you then change to the ipfire custom 
>>>> url then all the existing Toulouse blocklists stay in the directory 
>>>> on IPFire and so you end up with a huge number of category tick 
>>>> boxes, most of which are the old Toulouse ones, which are still 
>>>> available to select and it is not clear which ones are from 
>>>> Toulouse and which ones from IPFire.
>>>
>>> Yes, I got the same thing, too. I think this is a bug, too, because 
>>> otherwise you would have a lot of unused categories lying around 
>>> that will never be updated. You cannot even tell which ones are from 
>>> the current list and which ones from the old list.
>>>
>>> Long-term we could even consider to remove the Univ. Toulouse list 
>>> entirely and only have our own lists available which would make the 
>>> problem go away.
>>>
>>>> I think if the blocklist URL source is changed or a custom url is 
>>>> provided the first step should be to remove the old ones already 
>>>> existing.
>>>> That might be a problem because users can also create their own 
>>>> blocklists and I believe those go into the same directory.
>>>
>>> Good thought. We of course cannot delete the custom lists.
>>>
>>>> Without clearing out the old blocklists you end up with a huge 
>>>> number of checkboxes for lists but it is not clear what happens if 
>>>> there is a category that has the same name for the Toulouse list 
>>>> and the IPFire list such as gambling. I will have a look at that 
>>>> and see what happens.
>>>>
>>>> Not sure what the best approach to this is.
>>>
>>> I believe it is removing all old content.
>>>
>>>> Manually deleting all contents of the urlfilter/blacklists/ 
>>>> directory and then selecting the IPFire blocklist url for the 
>>>> custom url I end up with only the 8 categories from the IPFire list.
>>>>
>>>> I have tested some gambling sites from the IPFire list and the 
>>>> block worked on some. On others the site no longer exists so there 
>>>> is nothing to block or has been changed to an https site and in 
>>>> that case it went straight through. Also if I chose the http 
>>>> version of the link, it was automatically changed to https and went 
>>>> through without being blocked.
>>>
>>> The entire IPFire infrastructure always requires HTTPS. If you start 
>>> using HTTP, you will be automatically redirected. It is 2026 and we 
>>> don’t need to talk HTTP any more :)
>>
>> Some of the domains in the gambling list (maybe quite a lot) seem to 
>> only have an http access. If I tried https it came back with the fact 
>> that it couldn't find it.
>>
>>>
>>> I am glad to hear that the list is actually blocking. It would have 
>>> been bad if it didn’t. Now we have the big task to check out the 
>>> “quality” - however that can be determined. I think this is what 
>>> needs some time…
>>>
>>> In the meantime I have set up a small page on our website:
>>>
>>>    https://www.ipfire.org/dnsbl
>>>
>>> I would like to run this as a first-class project inside IPFire like 
>>> we are doing with IPFire Location. That means that we need to tell 
>>> people about what we are doing. Hopefully this page is a little start.
>>>
>>> Initially it has a couple of high-level bullet points about what we 
>>> are trying to achieve. I don’t think the text is very good, yet, but 
>>> it is the best I had in that moment. There is then also a list of 
>>> the lists that we currently offer. For each list, a detailed page 
>>> will tell you about the license, how many domains are listed, when 
>>> the last update has been, the sources and even there is a history 
>>> page that shows all the changes whenever they have happened.
>>>
>>> Finally there is a section that explains “How To Use?” the list 
>>> which I would love to extend to include AdGuard Plus and things like 
>>> that as well as Pi-Hole and whatever else could use the list. In a 
>>> later step we should go ahead and talk to any projects to include 
>>> our list(s) into their dropdown so that people can enable them nice 
>>> and easy.
>>>
>>> Behind the web page there is an API service that is running on the 
>>> host that is running the DNSBL. The frontend web app that is running 
>>> www.ipfire.org <http://www.ipfire.org/> is connecting to that API 
>>> service to fetch the current lists, any details and so on. That way, 
>>> we can split the logic and avoid creating a huge monolith of a web 
>>> app. This also means that page could be down a little as I am still 
>>> working on the entire thing and will frequently restart it.
>>>
>>> The API documentation is available here and the API is publicly 
>>> available: https://api.dnsbl.ipfire.org/docs
>>>
>>> The website/API allows to file reports for anything that does not 
>>> seem to be right on any of the lists. I would like to keep it as an 
>>> open process, however, long-term, this cannot cost us any time. In 
>>> the current stage, the reports are getting filed and that is about 
>>> it. I still need to build out some way for admins or moderators (I 
>>> am not sure what kind of roles I want to have here) to accept or 
>>> reject those reports.
>>>
>>> In case of us receiving a domain from a source list, I would rather 
>>> like to submit a report to upstream for them to de-list. That way, 
>>> we don’t have any admin to do and we are contributing back to other 
>>> list. That would be a very good thing to do. We cannot however throw 
>>> tons of emails at some random upstream projects without 
>>> co-ordinating this first. By not reporting upstream, we will 
>>> probably over time create large whitelists and I am not sure if that 
>>> is a good thing to do.
>>>
>>> Finally, there is a search box that can be used to find out if a 
>>> domain is listed on any of the lists.
>>>
>>>>> If you download and open any of the files, you will see a large 
>>>>> header that includes copyright information and lists all sources 
>>>>> that have been used to create the individual lists. This way we 
>>>>> ensure maximum transparency, comply with the terms of the 
>>>>> individual licenses of the source lists and give credit to the 
>>>>> people who help us to put together the most perfect list for our 
>>>>> users.
>>>>>
>>>>> I would like this to become a project that is not only being used 
>>>>> in IPFire. We can and will be compatible with other solutions like 
>>>>> AdGuard, PiHole so that people can use our lists if they would 
>>>>> like to even though they are not using IPFire. Hopefully, these 
>>>>> users will also feed back to us so that we can improve our lists 
>>>>> over time and make them one of the best options out there.
>>>>>
>>>>> All lists are available as a simple text file that lists the 
>>>>> domains. Then there is a hosts file available as well as a DNS 
>>>>> zone file and an RPZ file. Each list is individually available to 
>>>>> be used in squidGuard and there is a larger tarball available with 
>>>>> all lists that can be used in IPFire’s URL Filter. I am planning 
>>>>> to add Suricata/Snort signatures whenever I have time to do so. 
>>>>> Even though it is not a good idea to filter pornographic content 
>>>>> this way, I suppose that catching malware and blocking DoH are 
>>>>> good use-cases for an IPS. Time will tell…
>>>>>
>>>>> As a start, we will make these lists available in IPFire’s URL 
>>>>> Filter and collect some feedback about how we are doing. 
>>>>> Afterwards, we can see where else we can take this project.
>>>>>
>>>>> If you want to enable this on your system, simply add the URL to 
>>>>> your autoupdate.urls file like here:
>>>>>
>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f 
>>>>>
>>>> I also tested out adding the IPFire url to autoupdate.urls and that 
>>>> also worked fine for me.
>>>
>>> Very good. Should we include this already with Core Update 200? I 
>>> don’t think we would break anything, but we might already gain a 
>>> couple more people who are helping us to test this all?
>>
>> I think that would be a good idea.
>>
>>>
>>> The next step would be to build and test our DNS infrastructure. In 
>>> the “How To Use?” Section on the pages of the individual lists, you 
>>> can already see some instructions on how to use the lists as an RPZ. 
>>> In comparison to other “providers”, I would prefer if people would 
>>> be using DNS to fetch the lists. This is simply to push out updates 
>>> in a cheap way for us and also do it very regularly.
>>>
>>> Initially, clients will pull the entire list using AXFR. There is no 
>>> way around this as they need to have the data in the first place. 
>>> After that, clients will only need the changes. As you can see in 
>>> the history, the lists don’t actually change that often. Sometimes 
>>> only once a day and therefore downloading the entire list again 
>>> would be a huge waste of data, both on the client side, but also for 
>>> us hosting then.
>>>
>>> Some other providers update their lists “every 10 minutes”, and 
>>> there won't be any changes whatsoever. We don’t do that. We will 
>>> only export the lists again when they have actually changed. The 
>>> timestamps on the files that we offer using HTTPS can be checked by 
>>> clients so that they won’t re-download the list again if it has not 
>>> been changed. But using HTTPS still means that we would have to 
>>> re-download the entire list and not only the changes.
>>>
>>> Using DNS and IXFR will update the lists by only transferring a few 
>>> kilobytes and therefore we can have clients check once an hour if a 
>>> list has actually changed and only send out the raw changes. That 
>>> way, we will be able to serve millions of clients at very cheap cost 
>>> and they will always have a very up to date list.
>>>
>>> As far as I can see any DNS software that supports RPZs supports 
>>> AXFR/IXFR with exception of Knot Resolver which expects the zone to 
>>> be downloaded externally. There is a ticket for AXFR/IXFR support 
>>> (https://gitlab.nic.cz/knot/knot-resolver/-/issues/195).
>>>
>>> Initially, some of the lists have been *huge* which is why a simple 
>>> HTTP download is not feasible. The porn list was over 100 MiB. We 
>>> could have spent thousands on just traffic alone which I don’t have 
>>> for this kind of project. It would also be unnecessary money being 
>>> spent. There are simply better solutions out there. But then I built 
>>> something that basically tests the data that we are receiving from 
>>> upstream but simply checking if a listed domain still exists. The 
>>> result was very astonishing to me.
>>>
>>> So whenever someone adds a domain to the list, we will (eventually, 
>>> but not immediately) check if we can resolve the domain’s SOA 
>>> record. If not, we mark the domain as non-active and will no longer 
>>> include them in the exported data. This brought down the porn list 
>>> from just under 5 million domains to just 421k. On the sources page 
>>> (https://www.ipfire.org/dnsbl/lists/porn/sources) I am listing the 
>>> percentage of dead domains from each of them and the UT1 list has 
>>> 94% dead domains. Wow.
>>>
>>> If we cannot resolve the domain, neither can our users. So we would 
>>> otherwise fill the lists with tons of domains that simply could 
>>> never be reached. And if they cannot be reached, why would we block 
>>> them? We would waste bandwidth and a lot of memory on each single 
>>> client.
>>>
>>> The other sources have similarly high rations of dead domains. Most 
>>> of them are in the 50-80% range. Therefore I am happy that we are 
>>> doing some extra work here to give our users much better data for 
>>> their filtering.
>>
>> Removing all dead entries sounds like an excellent step.
>>
>> Regards,
>>
>> Adolf.
>>
>>>
>>> So, if you like, please go and check out the RPZ blocking with 
>>> Unbound. Instructions are on the page. I would be happy to hear how 
>>> this is turning out.
>>>
>>> Please let me know if there are any more questions, and I would be 
>>> glad to answer them.
>>>
>>> Happy New Year,
>>> -Michael
>>>
>>>>
>>>> Regards,
>>>> Adolf.
>>>>> This email is just a brain dump from me to this list. I would be 
>>>>> happy to answer any questions about implementation details, etc. 
>>>>> if people are interested. Right now, this email is long enough 
>>>>> already…
>>>>>
>>>>> All the best,
>>>>> -Michael
>>>>
>>>> -- 
>>>> Sent from my laptop
>>>
>>>
>>>
>>
>

-- 
Sent from my laptop

Re: Let's launch our own blocklists...

[Michael Tremer]

Hello Adolf,

This is a good find.

But if duckduckgo.com <http://duckduckgo.com/> is blocked, we will have to have a source somewhere that blocks that domain. Not only a sub-domain of it. Otherwise we have a bug somewhere.

This is most likely as the domain is listed here, but with some stuff afterwards:

  https://raw.githubusercontent.com/mtxadmin/ublock/refs/heads/master/hosts/_malware_typo

We strip everything after a # away because we consider it a comment. However, that causes that there is only a line with the domain left which will cause it being listed.

The # sign is used as some special character but at the same time it is being used for comments.

I will fix this and then refresh the list.

-Michael

> On 5 Jan 2026, at 11:31, Adolf Belka <adolf.belka@ipfire.org> wrote:
> 
> Hi Michael,
> 
> 
> On 05/01/2026 12:11, Adolf Belka wrote:
>> Hi Michael,
>> 
>> I have found that the malware list includes duckduckgo.com
>> 
> I have checked through the various sources used for the malware list.
> 
> The ShadowWhisperer (Tracking) list has improving.duckduckgo.com in its list. I suspect that this one is the one causing the problem.
> 
> The mtxadmin (_malware_typo) list has duckduckgo.com mentioned 3 times but not directly as a domain name - looks more like a reference.
> 
> Regards,
> 
> Adolf.
> 
> 
>> Regards,
>> Adolf.
>> 
>> 
>> On 02/01/2026 14:02, Adolf Belka wrote:
>>> Hi,
>>> 
>>> On 02/01/2026 12:09, Michael Tremer wrote:
>>>> Hello,
>>>> 
>>>>> On 30 Dec 2025, at 14:05, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>> 
>>>>> Hi Michael,
>>>>> 
>>>>> On 29/12/2025 13:05, Michael Tremer wrote:
>>>>>> Hello everyone,
>>>>>> 
>>>>>> I hope everyone had a great Christmas and a couple of quiet days to relax from all the stress that was the year 2025.
>>>>> Still relaxing.
>>>> 
>>>> Very good, so let’s have a strong start into 2026 now!
>>> 
>>> Starting next week, yes.
>>> 
>>>> 
>>>>>> Having a couple of quieter days, I have been working on a new, little (hopefully) side project that has probably been high up on our radar since the Shalla list has shut down in 2020, or maybe even earlier. The goal of the project is to provide good lists with categories of domain names which are usually used to block access to these domains.
>>>>>> 
>>>>>> I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists.
>>>>>> 
>>>>>> How did we get here?
>>>>>> 
>>>>>> As stated before, the URL filter feature in IPFire has the problem that there are not many good blocklists available any more. There used to be a couple more - most famously the Shalla list - but we are now down to a single list from the University of Toulouse. It is a great list, but it is not always the best fit for all users.
>>>>>> 
>>>>>> Then there has been talk about whether we could implement more blocking features into IPFire that don’t involve the proxy. Most famously blocking over DNS. The problem here remains a the blocking feature is only as good as the data that is fed into it. Some people have been putting forward a number of lists that were suitable for them, but they would not have replaced the blocking functionality as we know it. Their aim is to provide “one list for everything” but that is not what people usually want. It is targeted at a classic home user and the only separation that is being made is any adult/porn/NSFW content which usually is put into a separate list.
>>>>>> 
>>>>>> It would have been technically possible to include these lists and let the users decide, but that is not the aim of IPFire. We want to do the job for the user so that their job is getting easier. Including obscure lists that don’t have a clear outline of what they actually want to block (“bad content” is not a category) and passing the burden of figuring out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with cream on top is really not going to work. It is all confusing and will lead to a bad user experience.
>>>>>> 
>>>>>> An even bigger problem that is however completely impossible to solve is bad licensing of these lists. A user has asked the publisher of the HaGeZi list whether they could be included in IPFire and under what terms. The response was that the list is available under the terms of the GNU General Public License v3, but that does not seem to be true. The list contains data from various sources. Many of them are licensed under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public agreement that this data may be redistributed, there is a huge legal issue here. We would expose our users to potential copyright infringement which we cannot do under any circumstances. Furthermore many lists are available under a non-commercial license which excludes them from being used in any kind of business. Plenty of IPFire systems are running in businesses, if not even the vast majority.
>>>>>> 
>>>>>> In short, these lists are completely unusable for us. Apart from HaGeZi, I consider OISD to have the same problem.
>>>>>> 
>>>>>> Enough about all the things that are bad. Let’s talk about the new, good things:
>>>>>> 
>>>>>> Many blacklists on the internet are an amalgamation of other lists. These lists vary in quality with some of them being not that good and without a clear focus and others being excellent data. Since we don’t have the man power to start from scratch, I felt that we can copy the concept that HaGeZi and OISD have started and simply create a new list that is based on other lists at the beginning to have a good starting point. That way, we have much better control over what is going on these lists and we can shape and mould them as we need them. Most importantly, we don’t create a single lists, but many lists that have a clear focus and allow users to choose what they want to block and what not.
>>>>>> 
>>>>>> So the current experimental stage that I am in has these lists:
>>>>>> 
>>>>>>    * Ads
>>>>>>    * Dating
>>>>>>    * DoH
>>>>>>    * Gambling
>>>>>>    * Malware
>>>>>>    * Porn
>>>>>>    * Social
>>>>>>    * Violence
>>>>>> 
>>>>>> The categories have been determined by what source lists we have available with good data and are compatible with our chosen license CC BY-SA 4.0. This is the same license that we are using for the IPFire Location database, too.
>>>>>> 
>>>>>> The main use-cases for any kind of blocking are to comply with legal requirements in networks with children (i.e. schools) to remove any kind of pornographic content, sometimes block social media as well. Gambling and violence are commonly blocked, too. Even more common would be filtering advertising and any malicious content.
>>>>>> 
>>>>>> The latter is especially difficult because so many source lists throw phishing, spyware, malvertising, tracking and other things into the same bucket. Here this is currently all in the malware list which has therefore become quite large. I am not sure whether this will stay like this in the future or if we will have to make some adjustments, but that is exactly why this is now entering some larger testing.
>>>>>> 
>>>>>> What has been built so far? In order to put these lists together properly, track any data about where it is coming from, I have built a tool in Python available here:
>>>>>> 
>>>>>>    https://git.ipfire.org/?p=dnsbl.git;a=summary
>>>>>> 
>>>>>> This tool will automatically update all lists once an hour if there have been any changes and export them in various formats. The exported lists are available for download here:
>>>>>> 
>>>>>>    https://dnsbl.ipfire.org/lists/
>>>>> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the custom url works fine.
>>>>> 
>>>>> However you need to remember not to put the https:// at the front of the url otherwise the WUI page completes without any error messages but leaves an error message in the system logs saying
>>>>> 
>>>>> URL filter blacklist - ERROR: Not a valid URL filter blacklist
>>>>> 
>>>>> I found this out the hard way.
>>>> 
>>>> Oh yes, I forgot that there is a field on the web UI. If that does not accept https:// as a prefix, please file a bug and we will fix it.
>>> 
>>> I will confirm it and raise a bug.
>>> 
>>>> 
>>>>> The other thing I noticed is that if you already have the Toulouse University list downloaded and you then change to the ipfire custom url then all the existing Toulouse blocklists stay in the directory on IPFire and so you end up with a huge number of category tick boxes, most of which are the old Toulouse ones, which are still available to select and it is not clear which ones are from Toulouse and which ones from IPFire.
>>>> 
>>>> Yes, I got the same thing, too. I think this is a bug, too, because otherwise you would have a lot of unused categories lying around that will never be updated. You cannot even tell which ones are from the current list and which ones from the old list.
>>>> 
>>>> Long-term we could even consider to remove the Univ. Toulouse list entirely and only have our own lists available which would make the problem go away.
>>>> 
>>>>> I think if the blocklist URL source is changed or a custom url is provided the first step should be to remove the old ones already existing.
>>>>> That might be a problem because users can also create their own blocklists and I believe those go into the same directory.
>>>> 
>>>> Good thought. We of course cannot delete the custom lists.
>>>> 
>>>>> Without clearing out the old blocklists you end up with a huge number of checkboxes for lists but it is not clear what happens if there is a category that has the same name for the Toulouse list and the IPFire list such as gambling. I will have a look at that and see what happens.
>>>>> 
>>>>> Not sure what the best approach to this is.
>>>> 
>>>> I believe it is removing all old content.
>>>> 
>>>>> Manually deleting all contents of the urlfilter/blacklists/ directory and then selecting the IPFire blocklist url for the custom url I end up with only the 8 categories from the IPFire list.
>>>>> 
>>>>> I have tested some gambling sites from the IPFire list and the block worked on some. On others the site no longer exists so there is nothing to block or has been changed to an https site and in that case it went straight through. Also if I chose the http version of the link, it was automatically changed to https and went through without being blocked.
>>>> 
>>>> The entire IPFire infrastructure always requires HTTPS. If you start using HTTP, you will be automatically redirected. It is 2026 and we don’t need to talk HTTP any more :)
>>> 
>>> Some of the domains in the gambling list (maybe quite a lot) seem to only have an http access. If I tried https it came back with the fact that it couldn't find it.
>>> 
>>>> 
>>>> I am glad to hear that the list is actually blocking. It would have been bad if it didn’t. Now we have the big task to check out the “quality” - however that can be determined. I think this is what needs some time…
>>>> 
>>>> In the meantime I have set up a small page on our website:
>>>> 
>>>>    https://www.ipfire.org/dnsbl
>>>> 
>>>> I would like to run this as a first-class project inside IPFire like we are doing with IPFire Location. That means that we need to tell people about what we are doing. Hopefully this page is a little start.
>>>> 
>>>> Initially it has a couple of high-level bullet points about what we are trying to achieve. I don’t think the text is very good, yet, but it is the best I had in that moment. There is then also a list of the lists that we currently offer. For each list, a detailed page will tell you about the license, how many domains are listed, when the last update has been, the sources and even there is a history page that shows all the changes whenever they have happened.
>>>> 
>>>> Finally there is a section that explains “How To Use?” the list which I would love to extend to include AdGuard Plus and things like that as well as Pi-Hole and whatever else could use the list. In a later step we should go ahead and talk to any projects to include our list(s) into their dropdown so that people can enable them nice and easy.
>>>> 
>>>> Behind the web page there is an API service that is running on the host that is running the DNSBL. The frontend web app that is running www.ipfire.org <http://www.ipfire.org/> is connecting to that API service to fetch the current lists, any details and so on. That way, we can split the logic and avoid creating a huge monolith of a web app. This also means that page could be down a little as I am still working on the entire thing and will frequently restart it.
>>>> 
>>>> The API documentation is available here and the API is publicly available: https://api.dnsbl.ipfire.org/docs
>>>> 
>>>> The website/API allows to file reports for anything that does not seem to be right on any of the lists. I would like to keep it as an open process, however, long-term, this cannot cost us any time. In the current stage, the reports are getting filed and that is about it. I still need to build out some way for admins or moderators (I am not sure what kind of roles I want to have here) to accept or reject those reports.
>>>> 
>>>> In case of us receiving a domain from a source list, I would rather like to submit a report to upstream for them to de-list. That way, we don’t have any admin to do and we are contributing back to other list. That would be a very good thing to do. We cannot however throw tons of emails at some random upstream projects without co-ordinating this first. By not reporting upstream, we will probably over time create large whitelists and I am not sure if that is a good thing to do.
>>>> 
>>>> Finally, there is a search box that can be used to find out if a domain is listed on any of the lists.
>>>> 
>>>>>> If you download and open any of the files, you will see a large header that includes copyright information and lists all sources that have been used to create the individual lists. This way we ensure maximum transparency, comply with the terms of the individual licenses of the source lists and give credit to the people who help us to put together the most perfect list for our users.
>>>>>> 
>>>>>> I would like this to become a project that is not only being used in IPFire. We can and will be compatible with other solutions like AdGuard, PiHole so that people can use our lists if they would like to even though they are not using IPFire. Hopefully, these users will also feed back to us so that we can improve our lists over time and make them one of the best options out there.
>>>>>> 
>>>>>> All lists are available as a simple text file that lists the domains. Then there is a hosts file available as well as a DNS zone file and an RPZ file. Each list is individually available to be used in squidGuard and there is a larger tarball available with all lists that can be used in IPFire’s URL Filter. I am planning to add Suricata/Snort signatures whenever I have time to do so. Even though it is not a good idea to filter pornographic content this way, I suppose that catching malware and blocking DoH are good use-cases for an IPS. Time will tell…
>>>>>> 
>>>>>> As a start, we will make these lists available in IPFire’s URL Filter and collect some feedback about how we are doing. Afterwards, we can see where else we can take this project.
>>>>>> 
>>>>>> If you want to enable this on your system, simply add the URL to your autoupdate.urls file like here:
>>>>>> 
>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f
>>>>> I also tested out adding the IPFire url to autoupdate.urls and that also worked fine for me.
>>>> 
>>>> Very good. Should we include this already with Core Update 200? I don’t think we would break anything, but we might already gain a couple more people who are helping us to test this all?
>>> 
>>> I think that would be a good idea.
>>> 
>>>> 
>>>> The next step would be to build and test our DNS infrastructure. In the “How To Use?” Section on the pages of the individual lists, you can already see some instructions on how to use the lists as an RPZ. In comparison to other “providers”, I would prefer if people would be using DNS to fetch the lists. This is simply to push out updates in a cheap way for us and also do it very regularly.
>>>> 
>>>> Initially, clients will pull the entire list using AXFR. There is no way around this as they need to have the data in the first place. After that, clients will only need the changes. As you can see in the history, the lists don’t actually change that often. Sometimes only once a day and therefore downloading the entire list again would be a huge waste of data, both on the client side, but also for us hosting then.
>>>> 
>>>> Some other providers update their lists “every 10 minutes”, and there won't be any changes whatsoever. We don’t do that. We will only export the lists again when they have actually changed. The timestamps on the files that we offer using HTTPS can be checked by clients so that they won’t re-download the list again if it has not been changed. But using HTTPS still means that we would have to re-download the entire list and not only the changes.
>>>> 
>>>> Using DNS and IXFR will update the lists by only transferring a few kilobytes and therefore we can have clients check once an hour if a list has actually changed and only send out the raw changes. That way, we will be able to serve millions of clients at very cheap cost and they will always have a very up to date list.
>>>> 
>>>> As far as I can see any DNS software that supports RPZs supports AXFR/IXFR with exception of Knot Resolver which expects the zone to be downloaded externally. There is a ticket for AXFR/IXFR support (https://gitlab.nic.cz/knot/knot-resolver/-/issues/195).
>>>> 
>>>> Initially, some of the lists have been *huge* which is why a simple HTTP download is not feasible. The porn list was over 100 MiB. We could have spent thousands on just traffic alone which I don’t have for this kind of project. It would also be unnecessary money being spent. There are simply better solutions out there. But then I built something that basically tests the data that we are receiving from upstream but simply checking if a listed domain still exists. The result was very astonishing to me.
>>>> 
>>>> So whenever someone adds a domain to the list, we will (eventually, but not immediately) check if we can resolve the domain’s SOA record. If not, we mark the domain as non-active and will no longer include them in the exported data. This brought down the porn list from just under 5 million domains to just 421k. On the sources page (https://www.ipfire.org/dnsbl/lists/porn/sources) I am listing the percentage of dead domains from each of them and the UT1 list has 94% dead domains. Wow.
>>>> 
>>>> If we cannot resolve the domain, neither can our users. So we would otherwise fill the lists with tons of domains that simply could never be reached. And if they cannot be reached, why would we block them? We would waste bandwidth and a lot of memory on each single client.
>>>> 
>>>> The other sources have similarly high rations of dead domains. Most of them are in the 50-80% range. Therefore I am happy that we are doing some extra work here to give our users much better data for their filtering.
>>> 
>>> Removing all dead entries sounds like an excellent step.
>>> 
>>> Regards,
>>> 
>>> Adolf.
>>> 
>>>> 
>>>> So, if you like, please go and check out the RPZ blocking with Unbound. Instructions are on the page. I would be happy to hear how this is turning out.
>>>> 
>>>> Please let me know if there are any more questions, and I would be glad to answer them.
>>>> 
>>>> Happy New Year,
>>>> -Michael
>>>> 
>>>>> 
>>>>> Regards,
>>>>> Adolf.
>>>>>> This email is just a brain dump from me to this list. I would be happy to answer any questions about implementation details, etc. if people are interested. Right now, this email is long enough already…
>>>>>> 
>>>>>> All the best,
>>>>>> -Michael
>>>>> 
>>>>> -- 
>>>>> Sent from my laptop
>>>> 
>>>> 
>>>> 
>>> 
>> 
> 
> -- 
> Sent from my laptop
> 
> 

Re: Let's launch our own blocklists...

[Michael Tremer]

Good Morning Adolf,

I had a look at this problem yesterday and it seems that parsing the format is becoming a little bit difficult this way. Since this is only affecting very few domains, I have simply whitelisted them all manually and duckduckgo.com <http://duckduckgo.com/> and others should now be easily reachable again.

Please let me know if you have any more findings.

All the best,
-Michael

> On 5 Jan 2026, at 11:48, Michael Tremer <michael.tremer@ipfire.org> wrote:
> 
> Hello Adolf,
> 
> This is a good find.
> 
> But if duckduckgo.com <http://duckduckgo.com/> is blocked, we will have to have a source somewhere that blocks that domain. Not only a sub-domain of it. Otherwise we have a bug somewhere.
> 
> This is most likely as the domain is listed here, but with some stuff afterwards:
> 
>  https://raw.githubusercontent.com/mtxadmin/ublock/refs/heads/master/hosts/_malware_typo
> 
> We strip everything after a # away because we consider it a comment. However, that causes that there is only a line with the domain left which will cause it being listed.
> 
> The # sign is used as some special character but at the same time it is being used for comments.
> 
> I will fix this and then refresh the list.
> 
> -Michael
> 
>> On 5 Jan 2026, at 11:31, Adolf Belka <adolf.belka@ipfire.org> wrote:
>> 
>> Hi Michael,
>> 
>> 
>> On 05/01/2026 12:11, Adolf Belka wrote:
>>> Hi Michael,
>>> 
>>> I have found that the malware list includes duckduckgo.com
>>> 
>> I have checked through the various sources used for the malware list.
>> 
>> The ShadowWhisperer (Tracking) list has improving.duckduckgo.com in its list. I suspect that this one is the one causing the problem.
>> 
>> The mtxadmin (_malware_typo) list has duckduckgo.com mentioned 3 times but not directly as a domain name - looks more like a reference.
>> 
>> Regards,
>> 
>> Adolf.
>> 
>> 
>>> Regards,
>>> Adolf.
>>> 
>>> 
>>> On 02/01/2026 14:02, Adolf Belka wrote:
>>>> Hi,
>>>> 
>>>> On 02/01/2026 12:09, Michael Tremer wrote:
>>>>> Hello,
>>>>> 
>>>>>> On 30 Dec 2025, at 14:05, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>> 
>>>>>> Hi Michael,
>>>>>> 
>>>>>> On 29/12/2025 13:05, Michael Tremer wrote:
>>>>>>> Hello everyone,
>>>>>>> 
>>>>>>> I hope everyone had a great Christmas and a couple of quiet days to relax from all the stress that was the year 2025.
>>>>>> Still relaxing.
>>>>> 
>>>>> Very good, so let’s have a strong start into 2026 now!
>>>> 
>>>> Starting next week, yes.
>>>> 
>>>>> 
>>>>>>> Having a couple of quieter days, I have been working on a new, little (hopefully) side project that has probably been high up on our radar since the Shalla list has shut down in 2020, or maybe even earlier. The goal of the project is to provide good lists with categories of domain names which are usually used to block access to these domains.
>>>>>>> 
>>>>>>> I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists.
>>>>>>> 
>>>>>>> How did we get here?
>>>>>>> 
>>>>>>> As stated before, the URL filter feature in IPFire has the problem that there are not many good blocklists available any more. There used to be a couple more - most famously the Shalla list - but we are now down to a single list from the University of Toulouse. It is a great list, but it is not always the best fit for all users.
>>>>>>> 
>>>>>>> Then there has been talk about whether we could implement more blocking features into IPFire that don’t involve the proxy. Most famously blocking over DNS. The problem here remains a the blocking feature is only as good as the data that is fed into it. Some people have been putting forward a number of lists that were suitable for them, but they would not have replaced the blocking functionality as we know it. Their aim is to provide “one list for everything” but that is not what people usually want. It is targeted at a classic home user and the only separation that is being made is any adult/porn/NSFW content which usually is put into a separate list.
>>>>>>> 
>>>>>>> It would have been technically possible to include these lists and let the users decide, but that is not the aim of IPFire. We want to do the job for the user so that their job is getting easier. Including obscure lists that don’t have a clear outline of what they actually want to block (“bad content” is not a category) and passing the burden of figuring out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with cream on top is really not going to work. It is all confusing and will lead to a bad user experience.
>>>>>>> 
>>>>>>> An even bigger problem that is however completely impossible to solve is bad licensing of these lists. A user has asked the publisher of the HaGeZi list whether they could be included in IPFire and under what terms. The response was that the list is available under the terms of the GNU General Public License v3, but that does not seem to be true. The list contains data from various sources. Many of them are licensed under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public agreement that this data may be redistributed, there is a huge legal issue here. We would expose our users to potential copyright infringement which we cannot do under any circumstances. Furthermore many lists are available under a non-commercial license which excludes them from being used in any kind of business. Plenty of IPFire systems are running in businesses, if not even the vast majority.
>>>>>>> 
>>>>>>> In short, these lists are completely unusable for us. Apart from HaGeZi, I consider OISD to have the same problem.
>>>>>>> 
>>>>>>> Enough about all the things that are bad. Let’s talk about the new, good things:
>>>>>>> 
>>>>>>> Many blacklists on the internet are an amalgamation of other lists. These lists vary in quality with some of them being not that good and without a clear focus and others being excellent data. Since we don’t have the man power to start from scratch, I felt that we can copy the concept that HaGeZi and OISD have started and simply create a new list that is based on other lists at the beginning to have a good starting point. That way, we have much better control over what is going on these lists and we can shape and mould them as we need them. Most importantly, we don’t create a single lists, but many lists that have a clear focus and allow users to choose what they want to block and what not.
>>>>>>> 
>>>>>>> So the current experimental stage that I am in has these lists:
>>>>>>> 
>>>>>>>   * Ads
>>>>>>>   * Dating
>>>>>>>   * DoH
>>>>>>>   * Gambling
>>>>>>>   * Malware
>>>>>>>   * Porn
>>>>>>>   * Social
>>>>>>>   * Violence
>>>>>>> 
>>>>>>> The categories have been determined by what source lists we have available with good data and are compatible with our chosen license CC BY-SA 4.0. This is the same license that we are using for the IPFire Location database, too.
>>>>>>> 
>>>>>>> The main use-cases for any kind of blocking are to comply with legal requirements in networks with children (i.e. schools) to remove any kind of pornographic content, sometimes block social media as well. Gambling and violence are commonly blocked, too. Even more common would be filtering advertising and any malicious content.
>>>>>>> 
>>>>>>> The latter is especially difficult because so many source lists throw phishing, spyware, malvertising, tracking and other things into the same bucket. Here this is currently all in the malware list which has therefore become quite large. I am not sure whether this will stay like this in the future or if we will have to make some adjustments, but that is exactly why this is now entering some larger testing.
>>>>>>> 
>>>>>>> What has been built so far? In order to put these lists together properly, track any data about where it is coming from, I have built a tool in Python available here:
>>>>>>> 
>>>>>>>   https://git.ipfire.org/?p=dnsbl.git;a=summary
>>>>>>> 
>>>>>>> This tool will automatically update all lists once an hour if there have been any changes and export them in various formats. The exported lists are available for download here:
>>>>>>> 
>>>>>>>   https://dnsbl.ipfire.org/lists/
>>>>>> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the custom url works fine.
>>>>>> 
>>>>>> However you need to remember not to put the https:// at the front of the url otherwise the WUI page completes without any error messages but leaves an error message in the system logs saying
>>>>>> 
>>>>>> URL filter blacklist - ERROR: Not a valid URL filter blacklist
>>>>>> 
>>>>>> I found this out the hard way.
>>>>> 
>>>>> Oh yes, I forgot that there is a field on the web UI. If that does not accept https:// as a prefix, please file a bug and we will fix it.
>>>> 
>>>> I will confirm it and raise a bug.
>>>> 
>>>>> 
>>>>>> The other thing I noticed is that if you already have the Toulouse University list downloaded and you then change to the ipfire custom url then all the existing Toulouse blocklists stay in the directory on IPFire and so you end up with a huge number of category tick boxes, most of which are the old Toulouse ones, which are still available to select and it is not clear which ones are from Toulouse and which ones from IPFire.
>>>>> 
>>>>> Yes, I got the same thing, too. I think this is a bug, too, because otherwise you would have a lot of unused categories lying around that will never be updated. You cannot even tell which ones are from the current list and which ones from the old list.
>>>>> 
>>>>> Long-term we could even consider to remove the Univ. Toulouse list entirely and only have our own lists available which would make the problem go away.
>>>>> 
>>>>>> I think if the blocklist URL source is changed or a custom url is provided the first step should be to remove the old ones already existing.
>>>>>> That might be a problem because users can also create their own blocklists and I believe those go into the same directory.
>>>>> 
>>>>> Good thought. We of course cannot delete the custom lists.
>>>>> 
>>>>>> Without clearing out the old blocklists you end up with a huge number of checkboxes for lists but it is not clear what happens if there is a category that has the same name for the Toulouse list and the IPFire list such as gambling. I will have a look at that and see what happens.
>>>>>> 
>>>>>> Not sure what the best approach to this is.
>>>>> 
>>>>> I believe it is removing all old content.
>>>>> 
>>>>>> Manually deleting all contents of the urlfilter/blacklists/ directory and then selecting the IPFire blocklist url for the custom url I end up with only the 8 categories from the IPFire list.
>>>>>> 
>>>>>> I have tested some gambling sites from the IPFire list and the block worked on some. On others the site no longer exists so there is nothing to block or has been changed to an https site and in that case it went straight through. Also if I chose the http version of the link, it was automatically changed to https and went through without being blocked.
>>>>> 
>>>>> The entire IPFire infrastructure always requires HTTPS. If you start using HTTP, you will be automatically redirected. It is 2026 and we don’t need to talk HTTP any more :)
>>>> 
>>>> Some of the domains in the gambling list (maybe quite a lot) seem to only have an http access. If I tried https it came back with the fact that it couldn't find it.
>>>> 
>>>>> 
>>>>> I am glad to hear that the list is actually blocking. It would have been bad if it didn’t. Now we have the big task to check out the “quality” - however that can be determined. I think this is what needs some time…
>>>>> 
>>>>> In the meantime I have set up a small page on our website:
>>>>> 
>>>>>   https://www.ipfire.org/dnsbl
>>>>> 
>>>>> I would like to run this as a first-class project inside IPFire like we are doing with IPFire Location. That means that we need to tell people about what we are doing. Hopefully this page is a little start.
>>>>> 
>>>>> Initially it has a couple of high-level bullet points about what we are trying to achieve. I don’t think the text is very good, yet, but it is the best I had in that moment. There is then also a list of the lists that we currently offer. For each list, a detailed page will tell you about the license, how many domains are listed, when the last update has been, the sources and even there is a history page that shows all the changes whenever they have happened.
>>>>> 
>>>>> Finally there is a section that explains “How To Use?” the list which I would love to extend to include AdGuard Plus and things like that as well as Pi-Hole and whatever else could use the list. In a later step we should go ahead and talk to any projects to include our list(s) into their dropdown so that people can enable them nice and easy.
>>>>> 
>>>>> Behind the web page there is an API service that is running on the host that is running the DNSBL. The frontend web app that is running www.ipfire.org <http://www.ipfire.org/> is connecting to that API service to fetch the current lists, any details and so on. That way, we can split the logic and avoid creating a huge monolith of a web app. This also means that page could be down a little as I am still working on the entire thing and will frequently restart it.
>>>>> 
>>>>> The API documentation is available here and the API is publicly available: https://api.dnsbl.ipfire.org/docs
>>>>> 
>>>>> The website/API allows to file reports for anything that does not seem to be right on any of the lists. I would like to keep it as an open process, however, long-term, this cannot cost us any time. In the current stage, the reports are getting filed and that is about it. I still need to build out some way for admins or moderators (I am not sure what kind of roles I want to have here) to accept or reject those reports.
>>>>> 
>>>>> In case of us receiving a domain from a source list, I would rather like to submit a report to upstream for them to de-list. That way, we don’t have any admin to do and we are contributing back to other list. That would be a very good thing to do. We cannot however throw tons of emails at some random upstream projects without co-ordinating this first. By not reporting upstream, we will probably over time create large whitelists and I am not sure if that is a good thing to do.
>>>>> 
>>>>> Finally, there is a search box that can be used to find out if a domain is listed on any of the lists.
>>>>> 
>>>>>>> If you download and open any of the files, you will see a large header that includes copyright information and lists all sources that have been used to create the individual lists. This way we ensure maximum transparency, comply with the terms of the individual licenses of the source lists and give credit to the people who help us to put together the most perfect list for our users.
>>>>>>> 
>>>>>>> I would like this to become a project that is not only being used in IPFire. We can and will be compatible with other solutions like AdGuard, PiHole so that people can use our lists if they would like to even though they are not using IPFire. Hopefully, these users will also feed back to us so that we can improve our lists over time and make them one of the best options out there.
>>>>>>> 
>>>>>>> All lists are available as a simple text file that lists the domains. Then there is a hosts file available as well as a DNS zone file and an RPZ file. Each list is individually available to be used in squidGuard and there is a larger tarball available with all lists that can be used in IPFire’s URL Filter. I am planning to add Suricata/Snort signatures whenever I have time to do so. Even though it is not a good idea to filter pornographic content this way, I suppose that catching malware and blocking DoH are good use-cases for an IPS. Time will tell…
>>>>>>> 
>>>>>>> As a start, we will make these lists available in IPFire’s URL Filter and collect some feedback about how we are doing. Afterwards, we can see where else we can take this project.
>>>>>>> 
>>>>>>> If you want to enable this on your system, simply add the URL to your autoupdate.urls file like here:
>>>>>>> 
>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f
>>>>>> I also tested out adding the IPFire url to autoupdate.urls and that also worked fine for me.
>>>>> 
>>>>> Very good. Should we include this already with Core Update 200? I don’t think we would break anything, but we might already gain a couple more people who are helping us to test this all?
>>>> 
>>>> I think that would be a good idea.
>>>> 
>>>>> 
>>>>> The next step would be to build and test our DNS infrastructure. In the “How To Use?” Section on the pages of the individual lists, you can already see some instructions on how to use the lists as an RPZ. In comparison to other “providers”, I would prefer if people would be using DNS to fetch the lists. This is simply to push out updates in a cheap way for us and also do it very regularly.
>>>>> 
>>>>> Initially, clients will pull the entire list using AXFR. There is no way around this as they need to have the data in the first place. After that, clients will only need the changes. As you can see in the history, the lists don’t actually change that often. Sometimes only once a day and therefore downloading the entire list again would be a huge waste of data, both on the client side, but also for us hosting then.
>>>>> 
>>>>> Some other providers update their lists “every 10 minutes”, and there won't be any changes whatsoever. We don’t do that. We will only export the lists again when they have actually changed. The timestamps on the files that we offer using HTTPS can be checked by clients so that they won’t re-download the list again if it has not been changed. But using HTTPS still means that we would have to re-download the entire list and not only the changes.
>>>>> 
>>>>> Using DNS and IXFR will update the lists by only transferring a few kilobytes and therefore we can have clients check once an hour if a list has actually changed and only send out the raw changes. That way, we will be able to serve millions of clients at very cheap cost and they will always have a very up to date list.
>>>>> 
>>>>> As far as I can see any DNS software that supports RPZs supports AXFR/IXFR with exception of Knot Resolver which expects the zone to be downloaded externally. There is a ticket for AXFR/IXFR support (https://gitlab.nic.cz/knot/knot-resolver/-/issues/195).
>>>>> 
>>>>> Initially, some of the lists have been *huge* which is why a simple HTTP download is not feasible. The porn list was over 100 MiB. We could have spent thousands on just traffic alone which I don’t have for this kind of project. It would also be unnecessary money being spent. There are simply better solutions out there. But then I built something that basically tests the data that we are receiving from upstream but simply checking if a listed domain still exists. The result was very astonishing to me.
>>>>> 
>>>>> So whenever someone adds a domain to the list, we will (eventually, but not immediately) check if we can resolve the domain’s SOA record. If not, we mark the domain as non-active and will no longer include them in the exported data. This brought down the porn list from just under 5 million domains to just 421k. On the sources page (https://www.ipfire.org/dnsbl/lists/porn/sources) I am listing the percentage of dead domains from each of them and the UT1 list has 94% dead domains. Wow.
>>>>> 
>>>>> If we cannot resolve the domain, neither can our users. So we would otherwise fill the lists with tons of domains that simply could never be reached. And if they cannot be reached, why would we block them? We would waste bandwidth and a lot of memory on each single client.
>>>>> 
>>>>> The other sources have similarly high rations of dead domains. Most of them are in the 50-80% range. Therefore I am happy that we are doing some extra work here to give our users much better data for their filtering.
>>>> 
>>>> Removing all dead entries sounds like an excellent step.
>>>> 
>>>> Regards,
>>>> 
>>>> Adolf.
>>>> 
>>>>> 
>>>>> So, if you like, please go and check out the RPZ blocking with Unbound. Instructions are on the page. I would be happy to hear how this is turning out.
>>>>> 
>>>>> Please let me know if there are any more questions, and I would be glad to answer them.
>>>>> 
>>>>> Happy New Year,
>>>>> -Michael
>>>>> 
>>>>>> 
>>>>>> Regards,
>>>>>> Adolf.
>>>>>>> This email is just a brain dump from me to this list. I would be happy to answer any questions about implementation details, etc. if people are interested. Right now, this email is long enough already…
>>>>>>> 
>>>>>>> All the best,
>>>>>>> -Michael
>>>>>> 
>>>>>> -- 
>>>>>> Sent from my laptop
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> -- 
>> Sent from my laptop
>> 
>> 
> 

Re: Let's launch our own blocklists...

[Michael Tremer]

Hello everyone,

Over the past few weeks I have made significant progress on this all, and I think we're getting close to something the community will be really happy with. I'd love to get feedback from the team before we finalise things.

So what has happened?

First of all, the entire project has been renamed. DNSBL is not entirely what this is. Although the lists can be thrown into DNS, they have much more use outside of it that I thought we should simply go with DBL, short for Domain Blocklist. After all, we are only importing domains. The new home of the project therefore is https://www.ipfire.org/dbl

I have added a couple more lists that I thought interesting and I have added a couple more sources that I considered a good start. Hopefully, we will soon gather some more feedback on how well this is all holding up. My main focus has however been on the technology that will power this project.

One of the bigger challenges was to create Suricata rules from the lists. Initially I tried to create a ton of rules but since our lists are so large, this quickly became too complicated. I have now settled on using a feature that is only available in more recent versions of Suricata (I believe 7 and later), but since we are already on Suricata 8 in IPFire this won’t be a problem for us. All domains for each list are basically compiled into one massively large dataset and one single rule is referring to that dataset. This way, we won’t have the option to remove any false-positives, but at least Suricata and the GUI won’t starve a really bad death when loading millions of rules.

Suricata will now be able to use our rules to block access to any listed domains of each of the categories over DNS, HTTP, TLS or QUIC. Although I don’t expect many users to use Suricata to block porn or other things, this is a great backstop to enforce any policy like that. For example, if there is a user on the network who is trying to circumvent the DNS server that might filter out certain domains, even after getting an IP address resolved through other means, they won’t be able to open a TLS/QUIC connection or send a HTTP request to all blocked domains. Some people have said they were interested in blocking DNS-over-HTTPS and this is a perfect way to do this and actually be sure that any server that is being blocked on the list will actually be completely inaccessible.

Those Suricata rules are already available for testing in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=9eb8751487d23dd354a105c28bdbbb0398fe6e85

I have chosen various severities for the lists. If someone was to block advertising using DBL, this is fine, but not a very severe alert. If someone chooses to block malware and there is a system on the network trying to access those domains, this is an alert worth being investigated by an admin. Our new Suricata Reporter will show those violations in different colours based on the severity which helps to identify the right alerts to further investigate.

Formerly I have asked you to test the lists using URL Filter. Those rules are now available as well in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=db160694279a4b10378447f775dd536fdfcfb02a

I talked about a method to remove any dead domains from any sources which is a great way to keep our lists smaller. The pure size of them is a problem in so many ways. That check was however a little bit too ambitious and I had to make it a little bit less eager. Basically if we are in doubt, we need to still list the domain because it might be resolvable by a user.

  https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=bb5b6e33b731501d45dea293505f7d42a61d5ce7

So how else could we make the lists smaller without losing any actual data? Since we sometimes list a whole TLD (e.g. .xxx or .porn), there is very little point in listing any domains of this TLD. They will always be caught anyways. So I built a check that marks all domains that don’t need to be included on the exported lists because they will never be needed and was able to shrink the size of the lists by a lot again.

The website does not show this data, but the API returns the number of “subsumed” domains (I didn’t have a better name):

  curl https://api.dbl.ipfire.org/lists | jq .

The number shown would normally be added to the total number of domains and usually cuts the size of the list by 50-200%.

Those stats will now also be stored in a history table so that we will be able to track growth of all lists.

Furthermore, the application will now send email notifications for any incoming reports. This way, we will be able to stay in close touch with the reporters and keep them up to date on their submissions as well as inform moderators that there is something to have a look at.

The search has been refactored as well, so that we can show clearly whether something is blocked or not at one glance: https://www.ipfire.org/dbl/search?q=github.com. There is detailed information available on all domains and what happened to them. In case of GitHub.com, this seems to be blocked and unblocked by someone all of the time and we can see a clear audit trail of that: https://www.ipfire.org/dbl/lists/malware/domains/github.com

On the DNS front, I have added some metadata to the zones so that people can programmatically request some data, like when it has been last updated (in a human-friendly timestamp and not only the serial), license, description and so on:

  # dig +short ANY _info.ads.dbl.ipfire.org @primary.dbl.ipfire.org
  "total-domains=42226"
  "license=CC BY-SA 4.0"
  "updated-at=2026-01-20T22:17:02.409933+00:00"
  "description=Blocks domains used for ads, tracking, and ad delivery”

Now, I would like to hear more feedback from you. I know we've all been stretched thin lately, so I especially appreciate anyone who has time to review and provide input. Ideas, just say if you like it or not. Where this could go in the future?

Looking ahead, I would like us to start thinking about the RPZ feature that has been on the wishlist. IPFire DBL has been a bigger piece of work, and I think it's worth having a conversation about sustainability. Resources for this need to be allocated and paid for. Open source is about freedom, not free beer — and to keep building features like this, we will need to explore some funding options. I would be interested to hear any ideas you might have that could work for IPFire.

Please share your thoughts on the mailing list when you can — even a quick 'looks good' or 'I have concerns about X' is valuable. Public discussion helps everyone stay in the loop and contribute.

I am aiming to move forward with this in a week's time, so if you have input, now would be a good time to share it.

Best,
-Michael

> On 6 Jan 2026, at 10:20, Michael Tremer <michael.tremer@ipfire.org> wrote:
> 
> Good Morning Adolf,
> 
> I had a look at this problem yesterday and it seems that parsing the format is becoming a little bit difficult this way. Since this is only affecting very few domains, I have simply whitelisted them all manually and duckduckgo.com <http://duckduckgo.com/> and others should now be easily reachable again.
> 
> Please let me know if you have any more findings.
> 
> All the best,
> -Michael
> 
>> On 5 Jan 2026, at 11:48, Michael Tremer <michael.tremer@ipfire.org> wrote:
>> 
>> Hello Adolf,
>> 
>> This is a good find.
>> 
>> But if duckduckgo.com <http://duckduckgo.com/> is blocked, we will have to have a source somewhere that blocks that domain. Not only a sub-domain of it. Otherwise we have a bug somewhere.
>> 
>> This is most likely as the domain is listed here, but with some stuff afterwards:
>> 
>> https://raw.githubusercontent.com/mtxadmin/ublock/refs/heads/master/hosts/_malware_typo
>> 
>> We strip everything after a # away because we consider it a comment. However, that causes that there is only a line with the domain left which will cause it being listed.
>> 
>> The # sign is used as some special character but at the same time it is being used for comments.
>> 
>> I will fix this and then refresh the list.
>> 
>> -Michael
>> 
>>> On 5 Jan 2026, at 11:31, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>> 
>>> Hi Michael,
>>> 
>>> 
>>> On 05/01/2026 12:11, Adolf Belka wrote:
>>>> Hi Michael,
>>>> 
>>>> I have found that the malware list includes duckduckgo.com
>>>> 
>>> I have checked through the various sources used for the malware list.
>>> 
>>> The ShadowWhisperer (Tracking) list has improving.duckduckgo.com in its list. I suspect that this one is the one causing the problem.
>>> 
>>> The mtxadmin (_malware_typo) list has duckduckgo.com mentioned 3 times but not directly as a domain name - looks more like a reference.
>>> 
>>> Regards,
>>> 
>>> Adolf.
>>> 
>>> 
>>>> Regards,
>>>> Adolf.
>>>> 
>>>> 
>>>> On 02/01/2026 14:02, Adolf Belka wrote:
>>>>> Hi,
>>>>> 
>>>>> On 02/01/2026 12:09, Michael Tremer wrote:
>>>>>> Hello,
>>>>>> 
>>>>>>> On 30 Dec 2025, at 14:05, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>> 
>>>>>>> Hi Michael,
>>>>>>> 
>>>>>>> On 29/12/2025 13:05, Michael Tremer wrote:
>>>>>>>> Hello everyone,
>>>>>>>> 
>>>>>>>> I hope everyone had a great Christmas and a couple of quiet days to relax from all the stress that was the year 2025.
>>>>>>> Still relaxing.
>>>>>> 
>>>>>> Very good, so let’s have a strong start into 2026 now!
>>>>> 
>>>>> Starting next week, yes.
>>>>> 
>>>>>> 
>>>>>>>> Having a couple of quieter days, I have been working on a new, little (hopefully) side project that has probably been high up on our radar since the Shalla list has shut down in 2020, or maybe even earlier. The goal of the project is to provide good lists with categories of domain names which are usually used to block access to these domains.
>>>>>>>> 
>>>>>>>> I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists.
>>>>>>>> 
>>>>>>>> How did we get here?
>>>>>>>> 
>>>>>>>> As stated before, the URL filter feature in IPFire has the problem that there are not many good blocklists available any more. There used to be a couple more - most famously the Shalla list - but we are now down to a single list from the University of Toulouse. It is a great list, but it is not always the best fit for all users.
>>>>>>>> 
>>>>>>>> Then there has been talk about whether we could implement more blocking features into IPFire that don’t involve the proxy. Most famously blocking over DNS. The problem here remains a the blocking feature is only as good as the data that is fed into it. Some people have been putting forward a number of lists that were suitable for them, but they would not have replaced the blocking functionality as we know it. Their aim is to provide “one list for everything” but that is not what people usually want. It is targeted at a classic home user and the only separation that is being made is any adult/porn/NSFW content which usually is put into a separate list.
>>>>>>>> 
>>>>>>>> It would have been technically possible to include these lists and let the users decide, but that is not the aim of IPFire. We want to do the job for the user so that their job is getting easier. Including obscure lists that don’t have a clear outline of what they actually want to block (“bad content” is not a category) and passing the burden of figuring out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with cream on top is really not going to work. It is all confusing and will lead to a bad user experience.
>>>>>>>> 
>>>>>>>> An even bigger problem that is however completely impossible to solve is bad licensing of these lists. A user has asked the publisher of the HaGeZi list whether they could be included in IPFire and under what terms. The response was that the list is available under the terms of the GNU General Public License v3, but that does not seem to be true. The list contains data from various sources. Many of them are licensed under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public agreement that this data may be redistributed, there is a huge legal issue here. We would expose our users to potential copyright infringement which we cannot do under any circumstances. Furthermore many lists are available under a non-commercial license which excludes them from being used in any kind of business. Plenty of IPFire systems are running in businesses, if not even the vast majority.
>>>>>>>> 
>>>>>>>> In short, these lists are completely unusable for us. Apart from HaGeZi, I consider OISD to have the same problem.
>>>>>>>> 
>>>>>>>> Enough about all the things that are bad. Let’s talk about the new, good things:
>>>>>>>> 
>>>>>>>> Many blacklists on the internet are an amalgamation of other lists. These lists vary in quality with some of them being not that good and without a clear focus and others being excellent data. Since we don’t have the man power to start from scratch, I felt that we can copy the concept that HaGeZi and OISD have started and simply create a new list that is based on other lists at the beginning to have a good starting point. That way, we have much better control over what is going on these lists and we can shape and mould them as we need them. Most importantly, we don’t create a single lists, but many lists that have a clear focus and allow users to choose what they want to block and what not.
>>>>>>>> 
>>>>>>>> So the current experimental stage that I am in has these lists:
>>>>>>>> 
>>>>>>>>  * Ads
>>>>>>>>  * Dating
>>>>>>>>  * DoH
>>>>>>>>  * Gambling
>>>>>>>>  * Malware
>>>>>>>>  * Porn
>>>>>>>>  * Social
>>>>>>>>  * Violence
>>>>>>>> 
>>>>>>>> The categories have been determined by what source lists we have available with good data and are compatible with our chosen license CC BY-SA 4.0. This is the same license that we are using for the IPFire Location database, too.
>>>>>>>> 
>>>>>>>> The main use-cases for any kind of blocking are to comply with legal requirements in networks with children (i.e. schools) to remove any kind of pornographic content, sometimes block social media as well. Gambling and violence are commonly blocked, too. Even more common would be filtering advertising and any malicious content.
>>>>>>>> 
>>>>>>>> The latter is especially difficult because so many source lists throw phishing, spyware, malvertising, tracking and other things into the same bucket. Here this is currently all in the malware list which has therefore become quite large. I am not sure whether this will stay like this in the future or if we will have to make some adjustments, but that is exactly why this is now entering some larger testing.
>>>>>>>> 
>>>>>>>> What has been built so far? In order to put these lists together properly, track any data about where it is coming from, I have built a tool in Python available here:
>>>>>>>> 
>>>>>>>>  https://git.ipfire.org/?p=dnsbl.git;a=summary
>>>>>>>> 
>>>>>>>> This tool will automatically update all lists once an hour if there have been any changes and export them in various formats. The exported lists are available for download here:
>>>>>>>> 
>>>>>>>>  https://dnsbl.ipfire.org/lists/
>>>>>>> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the custom url works fine.
>>>>>>> 
>>>>>>> However you need to remember not to put the https:// at the front of the url otherwise the WUI page completes without any error messages but leaves an error message in the system logs saying
>>>>>>> 
>>>>>>> URL filter blacklist - ERROR: Not a valid URL filter blacklist
>>>>>>> 
>>>>>>> I found this out the hard way.
>>>>>> 
>>>>>> Oh yes, I forgot that there is a field on the web UI. If that does not accept https:// as a prefix, please file a bug and we will fix it.
>>>>> 
>>>>> I will confirm it and raise a bug.
>>>>> 
>>>>>> 
>>>>>>> The other thing I noticed is that if you already have the Toulouse University list downloaded and you then change to the ipfire custom url then all the existing Toulouse blocklists stay in the directory on IPFire and so you end up with a huge number of category tick boxes, most of which are the old Toulouse ones, which are still available to select and it is not clear which ones are from Toulouse and which ones from IPFire.
>>>>>> 
>>>>>> Yes, I got the same thing, too. I think this is a bug, too, because otherwise you would have a lot of unused categories lying around that will never be updated. You cannot even tell which ones are from the current list and which ones from the old list.
>>>>>> 
>>>>>> Long-term we could even consider to remove the Univ. Toulouse list entirely and only have our own lists available which would make the problem go away.
>>>>>> 
>>>>>>> I think if the blocklist URL source is changed or a custom url is provided the first step should be to remove the old ones already existing.
>>>>>>> That might be a problem because users can also create their own blocklists and I believe those go into the same directory.
>>>>>> 
>>>>>> Good thought. We of course cannot delete the custom lists.
>>>>>> 
>>>>>>> Without clearing out the old blocklists you end up with a huge number of checkboxes for lists but it is not clear what happens if there is a category that has the same name for the Toulouse list and the IPFire list such as gambling. I will have a look at that and see what happens.
>>>>>>> 
>>>>>>> Not sure what the best approach to this is.
>>>>>> 
>>>>>> I believe it is removing all old content.
>>>>>> 
>>>>>>> Manually deleting all contents of the urlfilter/blacklists/ directory and then selecting the IPFire blocklist url for the custom url I end up with only the 8 categories from the IPFire list.
>>>>>>> 
>>>>>>> I have tested some gambling sites from the IPFire list and the block worked on some. On others the site no longer exists so there is nothing to block or has been changed to an https site and in that case it went straight through. Also if I chose the http version of the link, it was automatically changed to https and went through without being blocked.
>>>>>> 
>>>>>> The entire IPFire infrastructure always requires HTTPS. If you start using HTTP, you will be automatically redirected. It is 2026 and we don’t need to talk HTTP any more :)
>>>>> 
>>>>> Some of the domains in the gambling list (maybe quite a lot) seem to only have an http access. If I tried https it came back with the fact that it couldn't find it.
>>>>> 
>>>>>> 
>>>>>> I am glad to hear that the list is actually blocking. It would have been bad if it didn’t. Now we have the big task to check out the “quality” - however that can be determined. I think this is what needs some time…
>>>>>> 
>>>>>> In the meantime I have set up a small page on our website:
>>>>>> 
>>>>>>  https://www.ipfire.org/dnsbl
>>>>>> 
>>>>>> I would like to run this as a first-class project inside IPFire like we are doing with IPFire Location. That means that we need to tell people about what we are doing. Hopefully this page is a little start.
>>>>>> 
>>>>>> Initially it has a couple of high-level bullet points about what we are trying to achieve. I don’t think the text is very good, yet, but it is the best I had in that moment. There is then also a list of the lists that we currently offer. For each list, a detailed page will tell you about the license, how many domains are listed, when the last update has been, the sources and even there is a history page that shows all the changes whenever they have happened.
>>>>>> 
>>>>>> Finally there is a section that explains “How To Use?” the list which I would love to extend to include AdGuard Plus and things like that as well as Pi-Hole and whatever else could use the list. In a later step we should go ahead and talk to any projects to include our list(s) into their dropdown so that people can enable them nice and easy.
>>>>>> 
>>>>>> Behind the web page there is an API service that is running on the host that is running the DNSBL. The frontend web app that is running www.ipfire.org <http://www.ipfire.org/> is connecting to that API service to fetch the current lists, any details and so on. That way, we can split the logic and avoid creating a huge monolith of a web app. This also means that page could be down a little as I am still working on the entire thing and will frequently restart it.
>>>>>> 
>>>>>> The API documentation is available here and the API is publicly available: https://api.dnsbl.ipfire.org/docs
>>>>>> 
>>>>>> The website/API allows to file reports for anything that does not seem to be right on any of the lists. I would like to keep it as an open process, however, long-term, this cannot cost us any time. In the current stage, the reports are getting filed and that is about it. I still need to build out some way for admins or moderators (I am not sure what kind of roles I want to have here) to accept or reject those reports.
>>>>>> 
>>>>>> In case of us receiving a domain from a source list, I would rather like to submit a report to upstream for them to de-list. That way, we don’t have any admin to do and we are contributing back to other list. That would be a very good thing to do. We cannot however throw tons of emails at some random upstream projects without co-ordinating this first. By not reporting upstream, we will probably over time create large whitelists and I am not sure if that is a good thing to do.
>>>>>> 
>>>>>> Finally, there is a search box that can be used to find out if a domain is listed on any of the lists.
>>>>>> 
>>>>>>>> If you download and open any of the files, you will see a large header that includes copyright information and lists all sources that have been used to create the individual lists. This way we ensure maximum transparency, comply with the terms of the individual licenses of the source lists and give credit to the people who help us to put together the most perfect list for our users.
>>>>>>>> 
>>>>>>>> I would like this to become a project that is not only being used in IPFire. We can and will be compatible with other solutions like AdGuard, PiHole so that people can use our lists if they would like to even though they are not using IPFire. Hopefully, these users will also feed back to us so that we can improve our lists over time and make them one of the best options out there.
>>>>>>>> 
>>>>>>>> All lists are available as a simple text file that lists the domains. Then there is a hosts file available as well as a DNS zone file and an RPZ file. Each list is individually available to be used in squidGuard and there is a larger tarball available with all lists that can be used in IPFire’s URL Filter. I am planning to add Suricata/Snort signatures whenever I have time to do so. Even though it is not a good idea to filter pornographic content this way, I suppose that catching malware and blocking DoH are good use-cases for an IPS. Time will tell…
>>>>>>>> 
>>>>>>>> As a start, we will make these lists available in IPFire’s URL Filter and collect some feedback about how we are doing. Afterwards, we can see where else we can take this project.
>>>>>>>> 
>>>>>>>> If you want to enable this on your system, simply add the URL to your autoupdate.urls file like here:
>>>>>>>> 
>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f
>>>>>>> I also tested out adding the IPFire url to autoupdate.urls and that also worked fine for me.
>>>>>> 
>>>>>> Very good. Should we include this already with Core Update 200? I don’t think we would break anything, but we might already gain a couple more people who are helping us to test this all?
>>>>> 
>>>>> I think that would be a good idea.
>>>>> 
>>>>>> 
>>>>>> The next step would be to build and test our DNS infrastructure. In the “How To Use?” Section on the pages of the individual lists, you can already see some instructions on how to use the lists as an RPZ. In comparison to other “providers”, I would prefer if people would be using DNS to fetch the lists. This is simply to push out updates in a cheap way for us and also do it very regularly.
>>>>>> 
>>>>>> Initially, clients will pull the entire list using AXFR. There is no way around this as they need to have the data in the first place. After that, clients will only need the changes. As you can see in the history, the lists don’t actually change that often. Sometimes only once a day and therefore downloading the entire list again would be a huge waste of data, both on the client side, but also for us hosting then.
>>>>>> 
>>>>>> Some other providers update their lists “every 10 minutes”, and there won't be any changes whatsoever. We don’t do that. We will only export the lists again when they have actually changed. The timestamps on the files that we offer using HTTPS can be checked by clients so that they won’t re-download the list again if it has not been changed. But using HTTPS still means that we would have to re-download the entire list and not only the changes.
>>>>>> 
>>>>>> Using DNS and IXFR will update the lists by only transferring a few kilobytes and therefore we can have clients check once an hour if a list has actually changed and only send out the raw changes. That way, we will be able to serve millions of clients at very cheap cost and they will always have a very up to date list.
>>>>>> 
>>>>>> As far as I can see any DNS software that supports RPZs supports AXFR/IXFR with exception of Knot Resolver which expects the zone to be downloaded externally. There is a ticket for AXFR/IXFR support (https://gitlab.nic.cz/knot/knot-resolver/-/issues/195).
>>>>>> 
>>>>>> Initially, some of the lists have been *huge* which is why a simple HTTP download is not feasible. The porn list was over 100 MiB. We could have spent thousands on just traffic alone which I don’t have for this kind of project. It would also be unnecessary money being spent. There are simply better solutions out there. But then I built something that basically tests the data that we are receiving from upstream but simply checking if a listed domain still exists. The result was very astonishing to me.
>>>>>> 
>>>>>> So whenever someone adds a domain to the list, we will (eventually, but not immediately) check if we can resolve the domain’s SOA record. If not, we mark the domain as non-active and will no longer include them in the exported data. This brought down the porn list from just under 5 million domains to just 421k. On the sources page (https://www.ipfire.org/dnsbl/lists/porn/sources) I am listing the percentage of dead domains from each of them and the UT1 list has 94% dead domains. Wow.
>>>>>> 
>>>>>> If we cannot resolve the domain, neither can our users. So we would otherwise fill the lists with tons of domains that simply could never be reached. And if they cannot be reached, why would we block them? We would waste bandwidth and a lot of memory on each single client.
>>>>>> 
>>>>>> The other sources have similarly high rations of dead domains. Most of them are in the 50-80% range. Therefore I am happy that we are doing some extra work here to give our users much better data for their filtering.
>>>>> 
>>>>> Removing all dead entries sounds like an excellent step.
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> Adolf.
>>>>> 
>>>>>> 
>>>>>> So, if you like, please go and check out the RPZ blocking with Unbound. Instructions are on the page. I would be happy to hear how this is turning out.
>>>>>> 
>>>>>> Please let me know if there are any more questions, and I would be glad to answer them.
>>>>>> 
>>>>>> Happy New Year,
>>>>>> -Michael
>>>>>> 
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Adolf.
>>>>>>>> This email is just a brain dump from me to this list. I would be happy to answer any questions about implementation details, etc. if people are interested. Right now, this email is long enough already…
>>>>>>>> 
>>>>>>>> All the best,
>>>>>>>> -Michael
>>>>>>> 
>>>>>>> -- 
>>>>>>> Sent from my laptop
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>>> -- 
>>> Sent from my laptop
>>> 
>>> 
>> 
> 

Re: Let's launch our own blocklists...

[Matthias Fischer]

On 22.01.2026 12:33, Michael Tremer wrote:
> Hello everyone,

Hi,

short feedback from me:

- I activated both the suricata (IPFire DBL - Domain Blocklist) - and
the URLfilter lists from 'dbl.ipfire.org'.

- I even took the 'smart-tv' domains from the IFire DBL blacklist and
copied/pasted them in my fritzbox filter lists.

Everything works as expected. Besides, the download of the IPFire
DBL-list loads a lot faster than the list from 'Univ. Toulouse'... ;-)

Functionality is good - no false positives or seen problems. Good work -
thanks!

Best
Matthias

> Over the past few weeks I have made significant progress on this all, and I think we're getting close to something the community will be really happy with. I'd love to get feedback from the team before we finalise things.
> 
> So what has happened?
> 
> First of all, the entire project has been renamed. DNSBL is not entirely what this is. Although the lists can be thrown into DNS, they have much more use outside of it that I thought we should simply go with DBL, short for Domain Blocklist. After all, we are only importing domains. The new home of the project therefore is https://www.ipfire.org/dbl
> 
> I have added a couple more lists that I thought interesting and I have added a couple more sources that I considered a good start. Hopefully, we will soon gather some more feedback on how well this is all holding up. My main focus has however been on the technology that will power this project.
> 
> One of the bigger challenges was to create Suricata rules from the lists. Initially I tried to create a ton of rules but since our lists are so large, this quickly became too complicated. I have now settled on using a feature that is only available in more recent versions of Suricata (I believe 7 and later), but since we are already on Suricata 8 in IPFire this won’t be a problem for us. All domains for each list are basically compiled into one massively large dataset and one single rule is referring to that dataset. This way, we won’t have the option to remove any false-positives, but at least Suricata and the GUI won’t starve a really bad death when loading millions of rules.
> 
> Suricata will now be able to use our rules to block access to any listed domains of each of the categories over DNS, HTTP, TLS or QUIC. Although I don’t expect many users to use Suricata to block porn or other things, this is a great backstop to enforce any policy like that. For example, if there is a user on the network who is trying to circumvent the DNS server that might filter out certain domains, even after getting an IP address resolved through other means, they won’t be able to open a TLS/QUIC connection or send a HTTP request to all blocked domains. Some people have said they were interested in blocking DNS-over-HTTPS and this is a perfect way to do this and actually be sure that any server that is being blocked on the list will actually be completely inaccessible.
> 
> Those Suricata rules are already available for testing in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=9eb8751487d23dd354a105c28bdbbb0398fe6e85
> 
> I have chosen various severities for the lists. If someone was to block advertising using DBL, this is fine, but not a very severe alert. If someone chooses to block malware and there is a system on the network trying to access those domains, this is an alert worth being investigated by an admin. Our new Suricata Reporter will show those violations in different colours based on the severity which helps to identify the right alerts to further investigate.
> 
> Formerly I have asked you to test the lists using URL Filter. Those rules are now available as well in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=db160694279a4b10378447f775dd536fdfcfb02a
> 
> I talked about a method to remove any dead domains from any sources which is a great way to keep our lists smaller. The pure size of them is a problem in so many ways. That check was however a little bit too ambitious and I had to make it a little bit less eager. Basically if we are in doubt, we need to still list the domain because it might be resolvable by a user.
> 
>   https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=bb5b6e33b731501d45dea293505f7d42a61d5ce7
> 
> So how else could we make the lists smaller without losing any actual data? Since we sometimes list a whole TLD (e.g. .xxx or .porn), there is very little point in listing any domains of this TLD. They will always be caught anyways. So I built a check that marks all domains that don’t need to be included on the exported lists because they will never be needed and was able to shrink the size of the lists by a lot again.
> 
> The website does not show this data, but the API returns the number of “subsumed” domains (I didn’t have a better name):
> 
>   curl https://api.dbl.ipfire.org/lists | jq .
> 
> The number shown would normally be added to the total number of domains and usually cuts the size of the list by 50-200%.
> 
> Those stats will now also be stored in a history table so that we will be able to track growth of all lists.
> 
> Furthermore, the application will now send email notifications for any incoming reports. This way, we will be able to stay in close touch with the reporters and keep them up to date on their submissions as well as inform moderators that there is something to have a look at.
> 
> The search has been refactored as well, so that we can show clearly whether something is blocked or not at one glance: https://www.ipfire.org/dbl/search?q=github.com. There is detailed information available on all domains and what happened to them. In case of GitHub.com, this seems to be blocked and unblocked by someone all of the time and we can see a clear audit trail of that: https://www.ipfire.org/dbl/lists/malware/domains/github.com
> 
> On the DNS front, I have added some metadata to the zones so that people can programmatically request some data, like when it has been last updated (in a human-friendly timestamp and not only the serial), license, description and so on:
> 
>   # dig +short ANY _info.ads.dbl.ipfire.org @primary.dbl.ipfire.org
>   "total-domains=42226"
>   "license=CC BY-SA 4.0"
>   "updated-at=2026-01-20T22:17:02.409933+00:00"
>   "description=Blocks domains used for ads, tracking, and ad delivery”
> 
> Now, I would like to hear more feedback from you. I know we've all been stretched thin lately, so I especially appreciate anyone who has time to review and provide input. Ideas, just say if you like it or not. Where this could go in the future?
> 
> Looking ahead, I would like us to start thinking about the RPZ feature that has been on the wishlist. IPFire DBL has been a bigger piece of work, and I think it's worth having a conversation about sustainability. Resources for this need to be allocated and paid for. Open source is about freedom, not free beer — and to keep building features like this, we will need to explore some funding options. I would be interested to hear any ideas you might have that could work for IPFire.
> 
> Please share your thoughts on the mailing list when you can — even a quick 'looks good' or 'I have concerns about X' is valuable. Public discussion helps everyone stay in the loop and contribute.
> 
> I am aiming to move forward with this in a week's time, so if you have input, now would be a good time to share it.
> 
> Best,
> -Michael
> 
>> On 6 Jan 2026, at 10:20, Michael Tremer <michael.tremer@ipfire.org> wrote:
>> 
>> Good Morning Adolf,
>> 
>> I had a look at this problem yesterday and it seems that parsing the format is becoming a little bit difficult this way. Since this is only affecting very few domains, I have simply whitelisted them all manually and duckduckgo.com <http://duckduckgo.com/> and others should now be easily reachable again.
>> 
>> Please let me know if you have any more findings.
>> 
>> All the best,
>> -Michael
>> 
>>> On 5 Jan 2026, at 11:48, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>> 
>>> Hello Adolf,
>>> 
>>> This is a good find.
>>> 
>>> But if duckduckgo.com <http://duckduckgo.com/> is blocked, we will have to have a source somewhere that blocks that domain. Not only a sub-domain of it. Otherwise we have a bug somewhere.
>>> 
>>> This is most likely as the domain is listed here, but with some stuff afterwards:
>>> 
>>> https://raw.githubusercontent.com/mtxadmin/ublock/refs/heads/master/hosts/_malware_typo
>>> 
>>> We strip everything after a # away because we consider it a comment. However, that causes that there is only a line with the domain left which will cause it being listed.
>>> 
>>> The # sign is used as some special character but at the same time it is being used for comments.
>>> 
>>> I will fix this and then refresh the list.
>>> 
>>> -Michael
>>> 
>>>> On 5 Jan 2026, at 11:31, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>> 
>>>> Hi Michael,
>>>> 
>>>> 
>>>> On 05/01/2026 12:11, Adolf Belka wrote:
>>>>> Hi Michael,
>>>>> 
>>>>> I have found that the malware list includes duckduckgo.com
>>>>> 
>>>> I have checked through the various sources used for the malware list.
>>>> 
>>>> The ShadowWhisperer (Tracking) list has improving.duckduckgo.com in its list. I suspect that this one is the one causing the problem.
>>>> 
>>>> The mtxadmin (_malware_typo) list has duckduckgo.com mentioned 3 times but not directly as a domain name - looks more like a reference.
>>>> 
>>>> Regards,
>>>> 
>>>> Adolf.
>>>> 
>>>> 
>>>>> Regards,
>>>>> Adolf.
>>>>> 
>>>>> 
>>>>> On 02/01/2026 14:02, Adolf Belka wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> On 02/01/2026 12:09, Michael Tremer wrote:
>>>>>>> Hello,
>>>>>>> 
>>>>>>>> On 30 Dec 2025, at 14:05, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>> 
>>>>>>>> Hi Michael,
>>>>>>>> 
>>>>>>>> On 29/12/2025 13:05, Michael Tremer wrote:
>>>>>>>>> Hello everyone,
>>>>>>>>> 
>>>>>>>>> I hope everyone had a great Christmas and a couple of quiet days to relax from all the stress that was the year 2025.
>>>>>>>> Still relaxing.
>>>>>>> 
>>>>>>> Very good, so let’s have a strong start into 2026 now!
>>>>>> 
>>>>>> Starting next week, yes.
>>>>>> 
>>>>>>> 
>>>>>>>>> Having a couple of quieter days, I have been working on a new, little (hopefully) side project that has probably been high up on our radar since the Shalla list has shut down in 2020, or maybe even earlier. The goal of the project is to provide good lists with categories of domain names which are usually used to block access to these domains.
>>>>>>>>> 
>>>>>>>>> I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists.
>>>>>>>>> 
>>>>>>>>> How did we get here?
>>>>>>>>> 
>>>>>>>>> As stated before, the URL filter feature in IPFire has the problem that there are not many good blocklists available any more. There used to be a couple more - most famously the Shalla list - but we are now down to a single list from the University of Toulouse. It is a great list, but it is not always the best fit for all users.
>>>>>>>>> 
>>>>>>>>> Then there has been talk about whether we could implement more blocking features into IPFire that don’t involve the proxy. Most famously blocking over DNS. The problem here remains a the blocking feature is only as good as the data that is fed into it. Some people have been putting forward a number of lists that were suitable for them, but they would not have replaced the blocking functionality as we know it. Their aim is to provide “one list for everything” but that is not what people usually want. It is targeted at a classic home user and the only separation that is being made is any adult/porn/NSFW content which usually is put into a separate list.
>>>>>>>>> 
>>>>>>>>> It would have been technically possible to include these lists and let the users decide, but that is not the aim of IPFire. We want to do the job for the user so that their job is getting easier. Including obscure lists that don’t have a clear outline of what they actually want to block (“bad content” is not a category) and passing the burden of figuring out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with cream on top is really not going to work. It is all confusing and will lead to a bad user experience.
>>>>>>>>> 
>>>>>>>>> An even bigger problem that is however completely impossible to solve is bad licensing of these lists. A user has asked the publisher of the HaGeZi list whether they could be included in IPFire and under what terms. The response was that the list is available under the terms of the GNU General Public License v3, but that does not seem to be true. The list contains data from various sources. Many of them are licensed under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public agreement that this data may be redistributed, there is a huge legal issue here. We would expose our users to potential copyright infringement which we cannot do under any circumstances. Furthermore many lists are available under a non-commercial license which excludes them from being used in any kind of business. Plenty of IPFire systems are running in businesses, if not even the vast majority.
>>>>>>>>> 
>>>>>>>>> In short, these lists are completely unusable for us. Apart from HaGeZi, I consider OISD to have the same problem.
>>>>>>>>> 
>>>>>>>>> Enough about all the things that are bad. Let’s talk about the new, good things:
>>>>>>>>> 
>>>>>>>>> Many blacklists on the internet are an amalgamation of other lists. These lists vary in quality with some of them being not that good and without a clear focus and others being excellent data. Since we don’t have the man power to start from scratch, I felt that we can copy the concept that HaGeZi and OISD have started and simply create a new list that is based on other lists at the beginning to have a good starting point. That way, we have much better control over what is going on these lists and we can shape and mould them as we need them. Most importantly, we don’t create a single lists, but many lists that have a clear focus and allow users to choose what they want to block and what not.
>>>>>>>>> 
>>>>>>>>> So the current experimental stage that I am in has these lists:
>>>>>>>>> 
>>>>>>>>>  * Ads
>>>>>>>>>  * Dating
>>>>>>>>>  * DoH
>>>>>>>>>  * Gambling
>>>>>>>>>  * Malware
>>>>>>>>>  * Porn
>>>>>>>>>  * Social
>>>>>>>>>  * Violence
>>>>>>>>> 
>>>>>>>>> The categories have been determined by what source lists we have available with good data and are compatible with our chosen license CC BY-SA 4.0. This is the same license that we are using for the IPFire Location database, too.
>>>>>>>>> 
>>>>>>>>> The main use-cases for any kind of blocking are to comply with legal requirements in networks with children (i.e. schools) to remove any kind of pornographic content, sometimes block social media as well. Gambling and violence are commonly blocked, too. Even more common would be filtering advertising and any malicious content.
>>>>>>>>> 
>>>>>>>>> The latter is especially difficult because so many source lists throw phishing, spyware, malvertising, tracking and other things into the same bucket. Here this is currently all in the malware list which has therefore become quite large. I am not sure whether this will stay like this in the future or if we will have to make some adjustments, but that is exactly why this is now entering some larger testing.
>>>>>>>>> 
>>>>>>>>> What has been built so far? In order to put these lists together properly, track any data about where it is coming from, I have built a tool in Python available here:
>>>>>>>>> 
>>>>>>>>>  https://git.ipfire.org/?p=dnsbl.git;a=summary
>>>>>>>>> 
>>>>>>>>> This tool will automatically update all lists once an hour if there have been any changes and export them in various formats. The exported lists are available for download here:
>>>>>>>>> 
>>>>>>>>>  https://dnsbl.ipfire.org/lists/
>>>>>>>> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the custom url works fine.
>>>>>>>> 
>>>>>>>> However you need to remember not to put the https:// at the front of the url otherwise the WUI page completes without any error messages but leaves an error message in the system logs saying
>>>>>>>> 
>>>>>>>> URL filter blacklist - ERROR: Not a valid URL filter blacklist
>>>>>>>> 
>>>>>>>> I found this out the hard way.
>>>>>>> 
>>>>>>> Oh yes, I forgot that there is a field on the web UI. If that does not accept https:// as a prefix, please file a bug and we will fix it.
>>>>>> 
>>>>>> I will confirm it and raise a bug.
>>>>>> 
>>>>>>> 
>>>>>>>> The other thing I noticed is that if you already have the Toulouse University list downloaded and you then change to the ipfire custom url then all the existing Toulouse blocklists stay in the directory on IPFire and so you end up with a huge number of category tick boxes, most of which are the old Toulouse ones, which are still available to select and it is not clear which ones are from Toulouse and which ones from IPFire.
>>>>>>> 
>>>>>>> Yes, I got the same thing, too. I think this is a bug, too, because otherwise you would have a lot of unused categories lying around that will never be updated. You cannot even tell which ones are from the current list and which ones from the old list.
>>>>>>> 
>>>>>>> Long-term we could even consider to remove the Univ. Toulouse list entirely and only have our own lists available which would make the problem go away.
>>>>>>> 
>>>>>>>> I think if the blocklist URL source is changed or a custom url is provided the first step should be to remove the old ones already existing.
>>>>>>>> That might be a problem because users can also create their own blocklists and I believe those go into the same directory.
>>>>>>> 
>>>>>>> Good thought. We of course cannot delete the custom lists.
>>>>>>> 
>>>>>>>> Without clearing out the old blocklists you end up with a huge number of checkboxes for lists but it is not clear what happens if there is a category that has the same name for the Toulouse list and the IPFire list such as gambling. I will have a look at that and see what happens.
>>>>>>>> 
>>>>>>>> Not sure what the best approach to this is.
>>>>>>> 
>>>>>>> I believe it is removing all old content.
>>>>>>> 
>>>>>>>> Manually deleting all contents of the urlfilter/blacklists/ directory and then selecting the IPFire blocklist url for the custom url I end up with only the 8 categories from the IPFire list.
>>>>>>>> 
>>>>>>>> I have tested some gambling sites from the IPFire list and the block worked on some. On others the site no longer exists so there is nothing to block or has been changed to an https site and in that case it went straight through. Also if I chose the http version of the link, it was automatically changed to https and went through without being blocked.
>>>>>>> 
>>>>>>> The entire IPFire infrastructure always requires HTTPS. If you start using HTTP, you will be automatically redirected. It is 2026 and we don’t need to talk HTTP any more :)
>>>>>> 
>>>>>> Some of the domains in the gambling list (maybe quite a lot) seem to only have an http access. If I tried https it came back with the fact that it couldn't find it.
>>>>>> 
>>>>>>> 
>>>>>>> I am glad to hear that the list is actually blocking. It would have been bad if it didn’t. Now we have the big task to check out the “quality” - however that can be determined. I think this is what needs some time…
>>>>>>> 
>>>>>>> In the meantime I have set up a small page on our website:
>>>>>>> 
>>>>>>>  https://www.ipfire.org/dnsbl
>>>>>>> 
>>>>>>> I would like to run this as a first-class project inside IPFire like we are doing with IPFire Location. That means that we need to tell people about what we are doing. Hopefully this page is a little start.
>>>>>>> 
>>>>>>> Initially it has a couple of high-level bullet points about what we are trying to achieve. I don’t think the text is very good, yet, but it is the best I had in that moment. There is then also a list of the lists that we currently offer. For each list, a detailed page will tell you about the license, how many domains are listed, when the last update has been, the sources and even there is a history page that shows all the changes whenever they have happened.
>>>>>>> 
>>>>>>> Finally there is a section that explains “How To Use?” the list which I would love to extend to include AdGuard Plus and things like that as well as Pi-Hole and whatever else could use the list. In a later step we should go ahead and talk to any projects to include our list(s) into their dropdown so that people can enable them nice and easy.
>>>>>>> 
>>>>>>> Behind the web page there is an API service that is running on the host that is running the DNSBL. The frontend web app that is running www.ipfire.org <http://www.ipfire.org/> is connecting to that API service to fetch the current lists, any details and so on. That way, we can split the logic and avoid creating a huge monolith of a web app. This also means that page could be down a little as I am still working on the entire thing and will frequently restart it.
>>>>>>> 
>>>>>>> The API documentation is available here and the API is publicly available: https://api.dnsbl.ipfire.org/docs
>>>>>>> 
>>>>>>> The website/API allows to file reports for anything that does not seem to be right on any of the lists. I would like to keep it as an open process, however, long-term, this cannot cost us any time. In the current stage, the reports are getting filed and that is about it. I still need to build out some way for admins or moderators (I am not sure what kind of roles I want to have here) to accept or reject those reports.
>>>>>>> 
>>>>>>> In case of us receiving a domain from a source list, I would rather like to submit a report to upstream for them to de-list. That way, we don’t have any admin to do and we are contributing back to other list. That would be a very good thing to do. We cannot however throw tons of emails at some random upstream projects without co-ordinating this first. By not reporting upstream, we will probably over time create large whitelists and I am not sure if that is a good thing to do.
>>>>>>> 
>>>>>>> Finally, there is a search box that can be used to find out if a domain is listed on any of the lists.
>>>>>>> 
>>>>>>>>> If you download and open any of the files, you will see a large header that includes copyright information and lists all sources that have been used to create the individual lists. This way we ensure maximum transparency, comply with the terms of the individual licenses of the source lists and give credit to the people who help us to put together the most perfect list for our users.
>>>>>>>>> 
>>>>>>>>> I would like this to become a project that is not only being used in IPFire. We can and will be compatible with other solutions like AdGuard, PiHole so that people can use our lists if they would like to even though they are not using IPFire. Hopefully, these users will also feed back to us so that we can improve our lists over time and make them one of the best options out there.
>>>>>>>>> 
>>>>>>>>> All lists are available as a simple text file that lists the domains. Then there is a hosts file available as well as a DNS zone file and an RPZ file. Each list is individually available to be used in squidGuard and there is a larger tarball available with all lists that can be used in IPFire’s URL Filter. I am planning to add Suricata/Snort signatures whenever I have time to do so. Even though it is not a good idea to filter pornographic content this way, I suppose that catching malware and blocking DoH are good use-cases for an IPS. Time will tell…
>>>>>>>>> 
>>>>>>>>> As a start, we will make these lists available in IPFire’s URL Filter and collect some feedback about how we are doing. Afterwards, we can see where else we can take this project.
>>>>>>>>> 
>>>>>>>>> If you want to enable this on your system, simply add the URL to your autoupdate.urls file like here:
>>>>>>>>> 
>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f
>>>>>>>> I also tested out adding the IPFire url to autoupdate.urls and that also worked fine for me.
>>>>>>> 
>>>>>>> Very good. Should we include this already with Core Update 200? I don’t think we would break anything, but we might already gain a couple more people who are helping us to test this all?
>>>>>> 
>>>>>> I think that would be a good idea.
>>>>>> 
>>>>>>> 
>>>>>>> The next step would be to build and test our DNS infrastructure. In the “How To Use?” Section on the pages of the individual lists, you can already see some instructions on how to use the lists as an RPZ. In comparison to other “providers”, I would prefer if people would be using DNS to fetch the lists. This is simply to push out updates in a cheap way for us and also do it very regularly.
>>>>>>> 
>>>>>>> Initially, clients will pull the entire list using AXFR. There is no way around this as they need to have the data in the first place. After that, clients will only need the changes. As you can see in the history, the lists don’t actually change that often. Sometimes only once a day and therefore downloading the entire list again would be a huge waste of data, both on the client side, but also for us hosting then.
>>>>>>> 
>>>>>>> Some other providers update their lists “every 10 minutes”, and there won't be any changes whatsoever. We don’t do that. We will only export the lists again when they have actually changed. The timestamps on the files that we offer using HTTPS can be checked by clients so that they won’t re-download the list again if it has not been changed. But using HTTPS still means that we would have to re-download the entire list and not only the changes.
>>>>>>> 
>>>>>>> Using DNS and IXFR will update the lists by only transferring a few kilobytes and therefore we can have clients check once an hour if a list has actually changed and only send out the raw changes. That way, we will be able to serve millions of clients at very cheap cost and they will always have a very up to date list.
>>>>>>> 
>>>>>>> As far as I can see any DNS software that supports RPZs supports AXFR/IXFR with exception of Knot Resolver which expects the zone to be downloaded externally. There is a ticket for AXFR/IXFR support (https://gitlab.nic.cz/knot/knot-resolver/-/issues/195).
>>>>>>> 
>>>>>>> Initially, some of the lists have been *huge* which is why a simple HTTP download is not feasible. The porn list was over 100 MiB. We could have spent thousands on just traffic alone which I don’t have for this kind of project. It would also be unnecessary money being spent. There are simply better solutions out there. But then I built something that basically tests the data that we are receiving from upstream but simply checking if a listed domain still exists. The result was very astonishing to me.
>>>>>>> 
>>>>>>> So whenever someone adds a domain to the list, we will (eventually, but not immediately) check if we can resolve the domain’s SOA record. If not, we mark the domain as non-active and will no longer include them in the exported data. This brought down the porn list from just under 5 million domains to just 421k. On the sources page (https://www.ipfire.org/dnsbl/lists/porn/sources) I am listing the percentage of dead domains from each of them and the UT1 list has 94% dead domains. Wow.
>>>>>>> 
>>>>>>> If we cannot resolve the domain, neither can our users. So we would otherwise fill the lists with tons of domains that simply could never be reached. And if they cannot be reached, why would we block them? We would waste bandwidth and a lot of memory on each single client.
>>>>>>> 
>>>>>>> The other sources have similarly high rations of dead domains. Most of them are in the 50-80% range. Therefore I am happy that we are doing some extra work here to give our users much better data for their filtering.
>>>>>> 
>>>>>> Removing all dead entries sounds like an excellent step.
>>>>>> 
>>>>>> Regards,
>>>>>> 
>>>>>> Adolf.
>>>>>> 
>>>>>>> 
>>>>>>> So, if you like, please go and check out the RPZ blocking with Unbound. Instructions are on the page. I would be happy to hear how this is turning out.
>>>>>>> 
>>>>>>> Please let me know if there are any more questions, and I would be glad to answer them.
>>>>>>> 
>>>>>>> Happy New Year,
>>>>>>> -Michael
>>>>>>> 
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> Adolf.
>>>>>>>>> This email is just a brain dump from me to this list. I would be happy to answer any questions about implementation details, etc. if people are interested. Right now, this email is long enough already…
>>>>>>>>> 
>>>>>>>>> All the best,
>>>>>>>>> -Michael
>>>>>>>> 
>>>>>>>> -- 
>>>>>>>> Sent from my laptop
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> -- 
>>>> Sent from my laptop
>>>> 
>>>> 
>>> 
>> 
> 
> 

Re: Let's launch our own blocklists...

[Michael Tremer]

Hello Matthias,

Thank you very much for testing IPFire DBL.

> On 23 Jan 2026, at 15:02, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
> 
> On 22.01.2026 12:33, Michael Tremer wrote:
>> Hello everyone,
> 
> Hi,
> 
> short feedback from me:
> 
> - I activated both the suricata (IPFire DBL - Domain Blocklist) - and
> the URLfilter lists from 'dbl.ipfire.org'.

This is an interesting case. What I didn’t manage to test yet is what happens when Suricata blocks the connection first. If URL Filter sees a domain that is being blocked it will either send you an error page if you are using HTTP, or simply close the connection if it is HTTPS. However, when Suricata comes first in the chain (and it will), it might close the connection because URL Filter has received the request. In the case of HTTPS this does not make any difference because the connection will be closed, but in the HTTP case you won’t see an error page any more and instead have the connection closed, too. You are basically losing the explicit error notification which is a little bit annoying.

We could have the same when we are doing the same with Unbound and DNS filtering. Potentially we would need to whitelist the local DNS resolver then, but how is Suricata supposed to know that the same categories are activated in both places?

> - I even took the 'smart-tv' domains from the IFire DBL blacklist and
> copied/pasted them in my fritzbox filter lists.

LOL Why not use IPFire to filter this as well?

> Everything works as expected. Besides, the download of the IPFire
> DBL-list loads a lot faster than the list from 'Univ. Toulouse'... ;-)

Yes, we don’t have much traffic on the server, yet.

> Functionality is good - no false positives or seen problems. Good work -
> thanks!

Nice. We need to distinguish a little between what is a technical issue and what is a false-positive/missing domain on the list. However, testing both at the same time is something we will all cope quite well with :)

-Michael

> Best
> Matthias
> 
>> Over the past few weeks I have made significant progress on this all, and I think we're getting close to something the community will be really happy with. I'd love to get feedback from the team before we finalise things.
>> 
>> So what has happened?
>> 
>> First of all, the entire project has been renamed. DNSBL is not entirely what this is. Although the lists can be thrown into DNS, they have much more use outside of it that I thought we should simply go with DBL, short for Domain Blocklist. After all, we are only importing domains. The new home of the project therefore is https://www.ipfire.org/dbl
>> 
>> I have added a couple more lists that I thought interesting and I have added a couple more sources that I considered a good start. Hopefully, we will soon gather some more feedback on how well this is all holding up. My main focus has however been on the technology that will power this project.
>> 
>> One of the bigger challenges was to create Suricata rules from the lists. Initially I tried to create a ton of rules but since our lists are so large, this quickly became too complicated. I have now settled on using a feature that is only available in more recent versions of Suricata (I believe 7 and later), but since we are already on Suricata 8 in IPFire this won’t be a problem for us. All domains for each list are basically compiled into one massively large dataset and one single rule is referring to that dataset. This way, we won’t have the option to remove any false-positives, but at least Suricata and the GUI won’t starve a really bad death when loading millions of rules.
>> 
>> Suricata will now be able to use our rules to block access to any listed domains of each of the categories over DNS, HTTP, TLS or QUIC. Although I don’t expect many users to use Suricata to block porn or other things, this is a great backstop to enforce any policy like that. For example, if there is a user on the network who is trying to circumvent the DNS server that might filter out certain domains, even after getting an IP address resolved through other means, they won’t be able to open a TLS/QUIC connection or send a HTTP request to all blocked domains. Some people have said they were interested in blocking DNS-over-HTTPS and this is a perfect way to do this and actually be sure that any server that is being blocked on the list will actually be completely inaccessible.
>> 
>> Those Suricata rules are already available for testing in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=9eb8751487d23dd354a105c28bdbbb0398fe6e85
>> 
>> I have chosen various severities for the lists. If someone was to block advertising using DBL, this is fine, but not a very severe alert. If someone chooses to block malware and there is a system on the network trying to access those domains, this is an alert worth being investigated by an admin. Our new Suricata Reporter will show those violations in different colours based on the severity which helps to identify the right alerts to further investigate.
>> 
>> Formerly I have asked you to test the lists using URL Filter. Those rules are now available as well in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=db160694279a4b10378447f775dd536fdfcfb02a
>> 
>> I talked about a method to remove any dead domains from any sources which is a great way to keep our lists smaller. The pure size of them is a problem in so many ways. That check was however a little bit too ambitious and I had to make it a little bit less eager. Basically if we are in doubt, we need to still list the domain because it might be resolvable by a user.
>> 
>>  https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=bb5b6e33b731501d45dea293505f7d42a61d5ce7
>> 
>> So how else could we make the lists smaller without losing any actual data? Since we sometimes list a whole TLD (e.g. .xxx or .porn), there is very little point in listing any domains of this TLD. They will always be caught anyways. So I built a check that marks all domains that don’t need to be included on the exported lists because they will never be needed and was able to shrink the size of the lists by a lot again.
>> 
>> The website does not show this data, but the API returns the number of “subsumed” domains (I didn’t have a better name):
>> 
>>  curl https://api.dbl.ipfire.org/lists | jq .
>> 
>> The number shown would normally be added to the total number of domains and usually cuts the size of the list by 50-200%.
>> 
>> Those stats will now also be stored in a history table so that we will be able to track growth of all lists.
>> 
>> Furthermore, the application will now send email notifications for any incoming reports. This way, we will be able to stay in close touch with the reporters and keep them up to date on their submissions as well as inform moderators that there is something to have a look at.
>> 
>> The search has been refactored as well, so that we can show clearly whether something is blocked or not at one glance: https://www.ipfire.org/dbl/search?q=github.com. There is detailed information available on all domains and what happened to them. In case of GitHub.com, this seems to be blocked and unblocked by someone all of the time and we can see a clear audit trail of that: https://www.ipfire.org/dbl/lists/malware/domains/github.com
>> 
>> On the DNS front, I have added some metadata to the zones so that people can programmatically request some data, like when it has been last updated (in a human-friendly timestamp and not only the serial), license, description and so on:
>> 
>>  # dig +short ANY _info.ads.dbl.ipfire.org @primary.dbl.ipfire.org
>>  "total-domains=42226"
>>  "license=CC BY-SA 4.0"
>>  "updated-at=2026-01-20T22:17:02.409933+00:00"
>>  "description=Blocks domains used for ads, tracking, and ad delivery”
>> 
>> Now, I would like to hear more feedback from you. I know we've all been stretched thin lately, so I especially appreciate anyone who has time to review and provide input. Ideas, just say if you like it or not. Where this could go in the future?
>> 
>> Looking ahead, I would like us to start thinking about the RPZ feature that has been on the wishlist. IPFire DBL has been a bigger piece of work, and I think it's worth having a conversation about sustainability. Resources for this need to be allocated and paid for. Open source is about freedom, not free beer — and to keep building features like this, we will need to explore some funding options. I would be interested to hear any ideas you might have that could work for IPFire.
>> 
>> Please share your thoughts on the mailing list when you can — even a quick 'looks good' or 'I have concerns about X' is valuable. Public discussion helps everyone stay in the loop and contribute.
>> 
>> I am aiming to move forward with this in a week's time, so if you have input, now would be a good time to share it.
>> 
>> Best,
>> -Michael
>> 
>>> On 6 Jan 2026, at 10:20, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>> 
>>> Good Morning Adolf,
>>> 
>>> I had a look at this problem yesterday and it seems that parsing the format is becoming a little bit difficult this way. Since this is only affecting very few domains, I have simply whitelisted them all manually and duckduckgo.com <http://duckduckgo.com/> and others should now be easily reachable again.
>>> 
>>> Please let me know if you have any more findings.
>>> 
>>> All the best,
>>> -Michael
>>> 
>>>> On 5 Jan 2026, at 11:48, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>>> 
>>>> Hello Adolf,
>>>> 
>>>> This is a good find.
>>>> 
>>>> But if duckduckgo.com <http://duckduckgo.com/> is blocked, we will have to have a source somewhere that blocks that domain. Not only a sub-domain of it. Otherwise we have a bug somewhere.
>>>> 
>>>> This is most likely as the domain is listed here, but with some stuff afterwards:
>>>> 
>>>> https://raw.githubusercontent.com/mtxadmin/ublock/refs/heads/master/hosts/_malware_typo
>>>> 
>>>> We strip everything after a # away because we consider it a comment. However, that causes that there is only a line with the domain left which will cause it being listed.
>>>> 
>>>> The # sign is used as some special character but at the same time it is being used for comments.
>>>> 
>>>> I will fix this and then refresh the list.
>>>> 
>>>> -Michael
>>>> 
>>>>> On 5 Jan 2026, at 11:31, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>> 
>>>>> Hi Michael,
>>>>> 
>>>>> 
>>>>> On 05/01/2026 12:11, Adolf Belka wrote:
>>>>>> Hi Michael,
>>>>>> 
>>>>>> I have found that the malware list includes duckduckgo.com
>>>>>> 
>>>>> I have checked through the various sources used for the malware list.
>>>>> 
>>>>> The ShadowWhisperer (Tracking) list has improving.duckduckgo.com in its list. I suspect that this one is the one causing the problem.
>>>>> 
>>>>> The mtxadmin (_malware_typo) list has duckduckgo.com mentioned 3 times but not directly as a domain name - looks more like a reference.
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> Adolf.
>>>>> 
>>>>> 
>>>>>> Regards,
>>>>>> Adolf.
>>>>>> 
>>>>>> 
>>>>>> On 02/01/2026 14:02, Adolf Belka wrote:
>>>>>>> Hi,
>>>>>>> 
>>>>>>> On 02/01/2026 12:09, Michael Tremer wrote:
>>>>>>>> Hello,
>>>>>>>> 
>>>>>>>>> On 30 Dec 2025, at 14:05, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>> 
>>>>>>>>> Hi Michael,
>>>>>>>>> 
>>>>>>>>> On 29/12/2025 13:05, Michael Tremer wrote:
>>>>>>>>>> Hello everyone,
>>>>>>>>>> 
>>>>>>>>>> I hope everyone had a great Christmas and a couple of quiet days to relax from all the stress that was the year 2025.
>>>>>>>>> Still relaxing.
>>>>>>>> 
>>>>>>>> Very good, so let’s have a strong start into 2026 now!
>>>>>>> 
>>>>>>> Starting next week, yes.
>>>>>>> 
>>>>>>>> 
>>>>>>>>>> Having a couple of quieter days, I have been working on a new, little (hopefully) side project that has probably been high up on our radar since the Shalla list has shut down in 2020, or maybe even earlier. The goal of the project is to provide good lists with categories of domain names which are usually used to block access to these domains.
>>>>>>>>>> 
>>>>>>>>>> I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists.
>>>>>>>>>> 
>>>>>>>>>> How did we get here?
>>>>>>>>>> 
>>>>>>>>>> As stated before, the URL filter feature in IPFire has the problem that there are not many good blocklists available any more. There used to be a couple more - most famously the Shalla list - but we are now down to a single list from the University of Toulouse. It is a great list, but it is not always the best fit for all users.
>>>>>>>>>> 
>>>>>>>>>> Then there has been talk about whether we could implement more blocking features into IPFire that don’t involve the proxy. Most famously blocking over DNS. The problem here remains a the blocking feature is only as good as the data that is fed into it. Some people have been putting forward a number of lists that were suitable for them, but they would not have replaced the blocking functionality as we know it. Their aim is to provide “one list for everything” but that is not what people usually want. It is targeted at a classic home user and the only separation that is being made is any adult/porn/NSFW content which usually is put into a separate list.
>>>>>>>>>> 
>>>>>>>>>> It would have been technically possible to include these lists and let the users decide, but that is not the aim of IPFire. We want to do the job for the user so that their job is getting easier. Including obscure lists that don’t have a clear outline of what they actually want to block (“bad content” is not a category) and passing the burden of figuring out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with cream on top is really not going to work. It is all confusing and will lead to a bad user experience.
>>>>>>>>>> 
>>>>>>>>>> An even bigger problem that is however completely impossible to solve is bad licensing of these lists. A user has asked the publisher of the HaGeZi list whether they could be included in IPFire and under what terms. The response was that the list is available under the terms of the GNU General Public License v3, but that does not seem to be true. The list contains data from various sources. Many of them are licensed under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public agreement that this data may be redistributed, there is a huge legal issue here. We would expose our users to potential copyright infringement which we cannot do under any circumstances. Furthermore many lists are available under a non-commercial license which excludes them from being used in any kind of business. Plenty of IPFire systems are running in businesses, if not even the vast majority.
>>>>>>>>>> 
>>>>>>>>>> In short, these lists are completely unusable for us. Apart from HaGeZi, I consider OISD to have the same problem.
>>>>>>>>>> 
>>>>>>>>>> Enough about all the things that are bad. Let’s talk about the new, good things:
>>>>>>>>>> 
>>>>>>>>>> Many blacklists on the internet are an amalgamation of other lists. These lists vary in quality with some of them being not that good and without a clear focus and others being excellent data. Since we don’t have the man power to start from scratch, I felt that we can copy the concept that HaGeZi and OISD have started and simply create a new list that is based on other lists at the beginning to have a good starting point. That way, we have much better control over what is going on these lists and we can shape and mould them as we need them. Most importantly, we don’t create a single lists, but many lists that have a clear focus and allow users to choose what they want to block and what not.
>>>>>>>>>> 
>>>>>>>>>> So the current experimental stage that I am in has these lists:
>>>>>>>>>> 
>>>>>>>>>> * Ads
>>>>>>>>>> * Dating
>>>>>>>>>> * DoH
>>>>>>>>>> * Gambling
>>>>>>>>>> * Malware
>>>>>>>>>> * Porn
>>>>>>>>>> * Social
>>>>>>>>>> * Violence
>>>>>>>>>> 
>>>>>>>>>> The categories have been determined by what source lists we have available with good data and are compatible with our chosen license CC BY-SA 4.0. This is the same license that we are using for the IPFire Location database, too.
>>>>>>>>>> 
>>>>>>>>>> The main use-cases for any kind of blocking are to comply with legal requirements in networks with children (i.e. schools) to remove any kind of pornographic content, sometimes block social media as well. Gambling and violence are commonly blocked, too. Even more common would be filtering advertising and any malicious content.
>>>>>>>>>> 
>>>>>>>>>> The latter is especially difficult because so many source lists throw phishing, spyware, malvertising, tracking and other things into the same bucket. Here this is currently all in the malware list which has therefore become quite large. I am not sure whether this will stay like this in the future or if we will have to make some adjustments, but that is exactly why this is now entering some larger testing.
>>>>>>>>>> 
>>>>>>>>>> What has been built so far? In order to put these lists together properly, track any data about where it is coming from, I have built a tool in Python available here:
>>>>>>>>>> 
>>>>>>>>>> https://git.ipfire.org/?p=dnsbl.git;a=summary
>>>>>>>>>> 
>>>>>>>>>> This tool will automatically update all lists once an hour if there have been any changes and export them in various formats. The exported lists are available for download here:
>>>>>>>>>> 
>>>>>>>>>> https://dnsbl.ipfire.org/lists/
>>>>>>>>> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the custom url works fine.
>>>>>>>>> 
>>>>>>>>> However you need to remember not to put the https:// at the front of the url otherwise the WUI page completes without any error messages but leaves an error message in the system logs saying
>>>>>>>>> 
>>>>>>>>> URL filter blacklist - ERROR: Not a valid URL filter blacklist
>>>>>>>>> 
>>>>>>>>> I found this out the hard way.
>>>>>>>> 
>>>>>>>> Oh yes, I forgot that there is a field on the web UI. If that does not accept https:// as a prefix, please file a bug and we will fix it.
>>>>>>> 
>>>>>>> I will confirm it and raise a bug.
>>>>>>> 
>>>>>>>> 
>>>>>>>>> The other thing I noticed is that if you already have the Toulouse University list downloaded and you then change to the ipfire custom url then all the existing Toulouse blocklists stay in the directory on IPFire and so you end up with a huge number of category tick boxes, most of which are the old Toulouse ones, which are still available to select and it is not clear which ones are from Toulouse and which ones from IPFire.
>>>>>>>> 
>>>>>>>> Yes, I got the same thing, too. I think this is a bug, too, because otherwise you would have a lot of unused categories lying around that will never be updated. You cannot even tell which ones are from the current list and which ones from the old list.
>>>>>>>> 
>>>>>>>> Long-term we could even consider to remove the Univ. Toulouse list entirely and only have our own lists available which would make the problem go away.
>>>>>>>> 
>>>>>>>>> I think if the blocklist URL source is changed or a custom url is provided the first step should be to remove the old ones already existing.
>>>>>>>>> That might be a problem because users can also create their own blocklists and I believe those go into the same directory.
>>>>>>>> 
>>>>>>>> Good thought. We of course cannot delete the custom lists.
>>>>>>>> 
>>>>>>>>> Without clearing out the old blocklists you end up with a huge number of checkboxes for lists but it is not clear what happens if there is a category that has the same name for the Toulouse list and the IPFire list such as gambling. I will have a look at that and see what happens.
>>>>>>>>> 
>>>>>>>>> Not sure what the best approach to this is.
>>>>>>>> 
>>>>>>>> I believe it is removing all old content.
>>>>>>>> 
>>>>>>>>> Manually deleting all contents of the urlfilter/blacklists/ directory and then selecting the IPFire blocklist url for the custom url I end up with only the 8 categories from the IPFire list.
>>>>>>>>> 
>>>>>>>>> I have tested some gambling sites from the IPFire list and the block worked on some. On others the site no longer exists so there is nothing to block or has been changed to an https site and in that case it went straight through. Also if I chose the http version of the link, it was automatically changed to https and went through without being blocked.
>>>>>>>> 
>>>>>>>> The entire IPFire infrastructure always requires HTTPS. If you start using HTTP, you will be automatically redirected. It is 2026 and we don’t need to talk HTTP any more :)
>>>>>>> 
>>>>>>> Some of the domains in the gambling list (maybe quite a lot) seem to only have an http access. If I tried https it came back with the fact that it couldn't find it.
>>>>>>> 
>>>>>>>> 
>>>>>>>> I am glad to hear that the list is actually blocking. It would have been bad if it didn’t. Now we have the big task to check out the “quality” - however that can be determined. I think this is what needs some time…
>>>>>>>> 
>>>>>>>> In the meantime I have set up a small page on our website:
>>>>>>>> 
>>>>>>>> https://www.ipfire.org/dnsbl
>>>>>>>> 
>>>>>>>> I would like to run this as a first-class project inside IPFire like we are doing with IPFire Location. That means that we need to tell people about what we are doing. Hopefully this page is a little start.
>>>>>>>> 
>>>>>>>> Initially it has a couple of high-level bullet points about what we are trying to achieve. I don’t think the text is very good, yet, but it is the best I had in that moment. There is then also a list of the lists that we currently offer. For each list, a detailed page will tell you about the license, how many domains are listed, when the last update has been, the sources and even there is a history page that shows all the changes whenever they have happened.
>>>>>>>> 
>>>>>>>> Finally there is a section that explains “How To Use?” the list which I would love to extend to include AdGuard Plus and things like that as well as Pi-Hole and whatever else could use the list. In a later step we should go ahead and talk to any projects to include our list(s) into their dropdown so that people can enable them nice and easy.
>>>>>>>> 
>>>>>>>> Behind the web page there is an API service that is running on the host that is running the DNSBL. The frontend web app that is running www.ipfire.org <http://www.ipfire.org/> is connecting to that API service to fetch the current lists, any details and so on. That way, we can split the logic and avoid creating a huge monolith of a web app. This also means that page could be down a little as I am still working on the entire thing and will frequently restart it.
>>>>>>>> 
>>>>>>>> The API documentation is available here and the API is publicly available: https://api.dnsbl.ipfire.org/docs
>>>>>>>> 
>>>>>>>> The website/API allows to file reports for anything that does not seem to be right on any of the lists. I would like to keep it as an open process, however, long-term, this cannot cost us any time. In the current stage, the reports are getting filed and that is about it. I still need to build out some way for admins or moderators (I am not sure what kind of roles I want to have here) to accept or reject those reports.
>>>>>>>> 
>>>>>>>> In case of us receiving a domain from a source list, I would rather like to submit a report to upstream for them to de-list. That way, we don’t have any admin to do and we are contributing back to other list. That would be a very good thing to do. We cannot however throw tons of emails at some random upstream projects without co-ordinating this first. By not reporting upstream, we will probably over time create large whitelists and I am not sure if that is a good thing to do.
>>>>>>>> 
>>>>>>>> Finally, there is a search box that can be used to find out if a domain is listed on any of the lists.
>>>>>>>> 
>>>>>>>>>> If you download and open any of the files, you will see a large header that includes copyright information and lists all sources that have been used to create the individual lists. This way we ensure maximum transparency, comply with the terms of the individual licenses of the source lists and give credit to the people who help us to put together the most perfect list for our users.
>>>>>>>>>> 
>>>>>>>>>> I would like this to become a project that is not only being used in IPFire. We can and will be compatible with other solutions like AdGuard, PiHole so that people can use our lists if they would like to even though they are not using IPFire. Hopefully, these users will also feed back to us so that we can improve our lists over time and make them one of the best options out there.
>>>>>>>>>> 
>>>>>>>>>> All lists are available as a simple text file that lists the domains. Then there is a hosts file available as well as a DNS zone file and an RPZ file. Each list is individually available to be used in squidGuard and there is a larger tarball available with all lists that can be used in IPFire’s URL Filter. I am planning to add Suricata/Snort signatures whenever I have time to do so. Even though it is not a good idea to filter pornographic content this way, I suppose that catching malware and blocking DoH are good use-cases for an IPS. Time will tell…
>>>>>>>>>> 
>>>>>>>>>> As a start, we will make these lists available in IPFire’s URL Filter and collect some feedback about how we are doing. Afterwards, we can see where else we can take this project.
>>>>>>>>>> 
>>>>>>>>>> If you want to enable this on your system, simply add the URL to your autoupdate.urls file like here:
>>>>>>>>>> 
>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f
>>>>>>>>> I also tested out adding the IPFire url to autoupdate.urls and that also worked fine for me.
>>>>>>>> 
>>>>>>>> Very good. Should we include this already with Core Update 200? I don’t think we would break anything, but we might already gain a couple more people who are helping us to test this all?
>>>>>>> 
>>>>>>> I think that would be a good idea.
>>>>>>> 
>>>>>>>> 
>>>>>>>> The next step would be to build and test our DNS infrastructure. In the “How To Use?” Section on the pages of the individual lists, you can already see some instructions on how to use the lists as an RPZ. In comparison to other “providers”, I would prefer if people would be using DNS to fetch the lists. This is simply to push out updates in a cheap way for us and also do it very regularly.
>>>>>>>> 
>>>>>>>> Initially, clients will pull the entire list using AXFR. There is no way around this as they need to have the data in the first place. After that, clients will only need the changes. As you can see in the history, the lists don’t actually change that often. Sometimes only once a day and therefore downloading the entire list again would be a huge waste of data, both on the client side, but also for us hosting then.
>>>>>>>> 
>>>>>>>> Some other providers update their lists “every 10 minutes”, and there won't be any changes whatsoever. We don’t do that. We will only export the lists again when they have actually changed. The timestamps on the files that we offer using HTTPS can be checked by clients so that they won’t re-download the list again if it has not been changed. But using HTTPS still means that we would have to re-download the entire list and not only the changes.
>>>>>>>> 
>>>>>>>> Using DNS and IXFR will update the lists by only transferring a few kilobytes and therefore we can have clients check once an hour if a list has actually changed and only send out the raw changes. That way, we will be able to serve millions of clients at very cheap cost and they will always have a very up to date list.
>>>>>>>> 
>>>>>>>> As far as I can see any DNS software that supports RPZs supports AXFR/IXFR with exception of Knot Resolver which expects the zone to be downloaded externally. There is a ticket for AXFR/IXFR support (https://gitlab.nic.cz/knot/knot-resolver/-/issues/195).
>>>>>>>> 
>>>>>>>> Initially, some of the lists have been *huge* which is why a simple HTTP download is not feasible. The porn list was over 100 MiB. We could have spent thousands on just traffic alone which I don’t have for this kind of project. It would also be unnecessary money being spent. There are simply better solutions out there. But then I built something that basically tests the data that we are receiving from upstream but simply checking if a listed domain still exists. The result was very astonishing to me.
>>>>>>>> 
>>>>>>>> So whenever someone adds a domain to the list, we will (eventually, but not immediately) check if we can resolve the domain’s SOA record. If not, we mark the domain as non-active and will no longer include them in the exported data. This brought down the porn list from just under 5 million domains to just 421k. On the sources page (https://www.ipfire.org/dnsbl/lists/porn/sources) I am listing the percentage of dead domains from each of them and the UT1 list has 94% dead domains. Wow.
>>>>>>>> 
>>>>>>>> If we cannot resolve the domain, neither can our users. So we would otherwise fill the lists with tons of domains that simply could never be reached. And if they cannot be reached, why would we block them? We would waste bandwidth and a lot of memory on each single client.
>>>>>>>> 
>>>>>>>> The other sources have similarly high rations of dead domains. Most of them are in the 50-80% range. Therefore I am happy that we are doing some extra work here to give our users much better data for their filtering.
>>>>>>> 
>>>>>>> Removing all dead entries sounds like an excellent step.
>>>>>>> 
>>>>>>> Regards,
>>>>>>> 
>>>>>>> Adolf.
>>>>>>> 
>>>>>>>> 
>>>>>>>> So, if you like, please go and check out the RPZ blocking with Unbound. Instructions are on the page. I would be happy to hear how this is turning out.
>>>>>>>> 
>>>>>>>> Please let me know if there are any more questions, and I would be glad to answer them.
>>>>>>>> 
>>>>>>>> Happy New Year,
>>>>>>>> -Michael
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> Adolf.
>>>>>>>>>> This email is just a brain dump from me to this list. I would be happy to answer any questions about implementation details, etc. if people are interested. Right now, this email is long enough already…
>>>>>>>>>> 
>>>>>>>>>> All the best,
>>>>>>>>>> -Michael
>>>>>>>>> 
>>>>>>>>> -- 
>>>>>>>>> Sent from my laptop
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> -- 
>>>>> Sent from my laptop
>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> 
> 
> 

Re: Let's launch our own blocklists...

[Matthias Fischer]

On 23.01.2026 17:39, Michael Tremer wrote:
> Hello Matthias,

Hi Michael,

I added the smart-tv rules to my Fritzbox because I had to migrate my
WiFi to the box due to the hostap-problems. After updating to Core 199
(see my mail from 07.01.2026), I couldn't access my repeater anymore and
the wireless log got flooded in no time with ~17000 entries and more.

Migrating was no problem - there are only a few wireless devices here,
but I wanted to test the lists with our TV. ;-)

> Thank you very much for testing IPFire DBL.
> 
>> On 23 Jan 2026, at 15:02, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>> 
>> On 22.01.2026 12:33, Michael Tremer wrote:
>>> Hello everyone,
>> 
>> Hi,
>> 
>> short feedback from me:
>> 
>> - I activated both the suricata (IPFire DBL - Domain Blocklist) - and
>> the URLfilter lists from 'dbl.ipfire.org'.
> 
> This is an interesting case. What I didn’t manage to test yet is what happens when Suricata blocks the connection first. If URL Filter sees a domain that is being blocked it will either send you an error page if you are using HTTP, or simply close the connection if it is HTTPS. However, when Suricata comes first in the chain (and it will), it might close the connection because URL Filter has received the request. In the case of HTTPS this does not make any difference because the connection will be closed, but in the HTTP case you won’t see an error page any more and instead have the connection closed, too. You are basically losing the explicit error notification which is a little bit annoying.
> 
> We could have the same when we are doing the same with Unbound and DNS filtering. Potentially we would need to whitelist the local DNS resolver then, but how is Suricata supposed to know that the same categories are activated in both places?
> 
>> - I even took the 'smart-tv' domains from the IFire DBL blacklist and
>> copied/pasted them in my fritzbox filter lists.
> 
> LOL Why not use IPFire to filter this as well?
> 
>> Everything works as expected. Besides, the download of the IPFire
>> DBL-list loads a lot faster than the list from 'Univ. Toulouse'... ;-)
> 
> Yes, we don’t have much traffic on the server, yet.
> 
>> Functionality is good - no false positives or seen problems. Good work -
>> thanks!
> 
> Nice. We need to distinguish a little between what is a technical issue and what is a false-positive/missing domain on the list. However, testing both at the same time is something we will all cope quite well with :)
> 
> -Michael
> 
>> Best
>> Matthias
>> 
>>> Over the past few weeks I have made significant progress on this all, and I think we're getting close to something the community will be really happy with. I'd love to get feedback from the team before we finalise things.
>>> 
>>> So what has happened?
>>> 
>>> First of all, the entire project has been renamed. DNSBL is not entirely what this is. Although the lists can be thrown into DNS, they have much more use outside of it that I thought we should simply go with DBL, short for Domain Blocklist. After all, we are only importing domains. The new home of the project therefore is https://www.ipfire.org/dbl
>>> 
>>> I have added a couple more lists that I thought interesting and I have added a couple more sources that I considered a good start. Hopefully, we will soon gather some more feedback on how well this is all holding up. My main focus has however been on the technology that will power this project.
>>> 
>>> One of the bigger challenges was to create Suricata rules from the lists. Initially I tried to create a ton of rules but since our lists are so large, this quickly became too complicated. I have now settled on using a feature that is only available in more recent versions of Suricata (I believe 7 and later), but since we are already on Suricata 8 in IPFire this won’t be a problem for us. All domains for each list are basically compiled into one massively large dataset and one single rule is referring to that dataset. This way, we won’t have the option to remove any false-positives, but at least Suricata and the GUI won’t starve a really bad death when loading millions of rules.
>>> 
>>> Suricata will now be able to use our rules to block access to any listed domains of each of the categories over DNS, HTTP, TLS or QUIC. Although I don’t expect many users to use Suricata to block porn or other things, this is a great backstop to enforce any policy like that. For example, if there is a user on the network who is trying to circumvent the DNS server that might filter out certain domains, even after getting an IP address resolved through other means, they won’t be able to open a TLS/QUIC connection or send a HTTP request to all blocked domains. Some people have said they were interested in blocking DNS-over-HTTPS and this is a perfect way to do this and actually be sure that any server that is being blocked on the list will actually be completely inaccessible.
>>> 
>>> Those Suricata rules are already available for testing in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=9eb8751487d23dd354a105c28bdbbb0398fe6e85
>>> 
>>> I have chosen various severities for the lists. If someone was to block advertising using DBL, this is fine, but not a very severe alert. If someone chooses to block malware and there is a system on the network trying to access those domains, this is an alert worth being investigated by an admin. Our new Suricata Reporter will show those violations in different colours based on the severity which helps to identify the right alerts to further investigate.
>>> 
>>> Formerly I have asked you to test the lists using URL Filter. Those rules are now available as well in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=db160694279a4b10378447f775dd536fdfcfb02a
>>> 
>>> I talked about a method to remove any dead domains from any sources which is a great way to keep our lists smaller. The pure size of them is a problem in so many ways. That check was however a little bit too ambitious and I had to make it a little bit less eager. Basically if we are in doubt, we need to still list the domain because it might be resolvable by a user.
>>> 
>>>  https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=bb5b6e33b731501d45dea293505f7d42a61d5ce7
>>> 
>>> So how else could we make the lists smaller without losing any actual data? Since we sometimes list a whole TLD (e.g. .xxx or .porn), there is very little point in listing any domains of this TLD. They will always be caught anyways. So I built a check that marks all domains that don’t need to be included on the exported lists because they will never be needed and was able to shrink the size of the lists by a lot again.
>>> 
>>> The website does not show this data, but the API returns the number of “subsumed” domains (I didn’t have a better name):
>>> 
>>>  curl https://api.dbl.ipfire.org/lists | jq .
>>> 
>>> The number shown would normally be added to the total number of domains and usually cuts the size of the list by 50-200%.
>>> 
>>> Those stats will now also be stored in a history table so that we will be able to track growth of all lists.
>>> 
>>> Furthermore, the application will now send email notifications for any incoming reports. This way, we will be able to stay in close touch with the reporters and keep them up to date on their submissions as well as inform moderators that there is something to have a look at.
>>> 
>>> The search has been refactored as well, so that we can show clearly whether something is blocked or not at one glance: https://www.ipfire.org/dbl/search?q=github.com. There is detailed information available on all domains and what happened to them. In case of GitHub.com, this seems to be blocked and unblocked by someone all of the time and we can see a clear audit trail of that: https://www.ipfire.org/dbl/lists/malware/domains/github.com
>>> 
>>> On the DNS front, I have added some metadata to the zones so that people can programmatically request some data, like when it has been last updated (in a human-friendly timestamp and not only the serial), license, description and so on:
>>> 
>>>  # dig +short ANY _info.ads.dbl.ipfire.org @primary.dbl.ipfire.org
>>>  "total-domains=42226"
>>>  "license=CC BY-SA 4.0"
>>>  "updated-at=2026-01-20T22:17:02.409933+00:00"
>>>  "description=Blocks domains used for ads, tracking, and ad delivery”
>>> 
>>> Now, I would like to hear more feedback from you. I know we've all been stretched thin lately, so I especially appreciate anyone who has time to review and provide input. Ideas, just say if you like it or not. Where this could go in the future?
>>> 
>>> Looking ahead, I would like us to start thinking about the RPZ feature that has been on the wishlist. IPFire DBL has been a bigger piece of work, and I think it's worth having a conversation about sustainability. Resources for this need to be allocated and paid for. Open source is about freedom, not free beer — and to keep building features like this, we will need to explore some funding options. I would be interested to hear any ideas you might have that could work for IPFire.
>>> 
>>> Please share your thoughts on the mailing list when you can — even a quick 'looks good' or 'I have concerns about X' is valuable. Public discussion helps everyone stay in the loop and contribute.
>>> 
>>> I am aiming to move forward with this in a week's time, so if you have input, now would be a good time to share it.
>>> 
>>> Best,
>>> -Michael
>>> 
>>>> On 6 Jan 2026, at 10:20, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>>> 
>>>> Good Morning Adolf,
>>>> 
>>>> I had a look at this problem yesterday and it seems that parsing the format is becoming a little bit difficult this way. Since this is only affecting very few domains, I have simply whitelisted them all manually and duckduckgo.com <http://duckduckgo.com/> and others should now be easily reachable again.
>>>> 
>>>> Please let me know if you have any more findings.
>>>> 
>>>> All the best,
>>>> -Michael
>>>> 
>>>>> On 5 Jan 2026, at 11:48, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>>>> 
>>>>> Hello Adolf,
>>>>> 
>>>>> This is a good find.
>>>>> 
>>>>> But if duckduckgo.com <http://duckduckgo.com/> is blocked, we will have to have a source somewhere that blocks that domain. Not only a sub-domain of it. Otherwise we have a bug somewhere.
>>>>> 
>>>>> This is most likely as the domain is listed here, but with some stuff afterwards:
>>>>> 
>>>>> https://raw.githubusercontent.com/mtxadmin/ublock/refs/heads/master/hosts/_malware_typo
>>>>> 
>>>>> We strip everything after a # away because we consider it a comment. However, that causes that there is only a line with the domain left which will cause it being listed.
>>>>> 
>>>>> The # sign is used as some special character but at the same time it is being used for comments.
>>>>> 
>>>>> I will fix this and then refresh the list.
>>>>> 
>>>>> -Michael
>>>>> 
>>>>>> On 5 Jan 2026, at 11:31, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>> 
>>>>>> Hi Michael,
>>>>>> 
>>>>>> 
>>>>>> On 05/01/2026 12:11, Adolf Belka wrote:
>>>>>>> Hi Michael,
>>>>>>> 
>>>>>>> I have found that the malware list includes duckduckgo.com
>>>>>>> 
>>>>>> I have checked through the various sources used for the malware list.
>>>>>> 
>>>>>> The ShadowWhisperer (Tracking) list has improving.duckduckgo.com in its list. I suspect that this one is the one causing the problem.
>>>>>> 
>>>>>> The mtxadmin (_malware_typo) list has duckduckgo.com mentioned 3 times but not directly as a domain name - looks more like a reference.
>>>>>> 
>>>>>> Regards,
>>>>>> 
>>>>>> Adolf.
>>>>>> 
>>>>>> 
>>>>>>> Regards,
>>>>>>> Adolf.
>>>>>>> 
>>>>>>> 
>>>>>>> On 02/01/2026 14:02, Adolf Belka wrote:
>>>>>>>> Hi,
>>>>>>>> 
>>>>>>>> On 02/01/2026 12:09, Michael Tremer wrote:
>>>>>>>>> Hello,
>>>>>>>>> 
>>>>>>>>>> On 30 Dec 2025, at 14:05, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi Michael,
>>>>>>>>>> 
>>>>>>>>>> On 29/12/2025 13:05, Michael Tremer wrote:
>>>>>>>>>>> Hello everyone,
>>>>>>>>>>> 
>>>>>>>>>>> I hope everyone had a great Christmas and a couple of quiet days to relax from all the stress that was the year 2025.
>>>>>>>>>> Still relaxing.
>>>>>>>>> 
>>>>>>>>> Very good, so let’s have a strong start into 2026 now!
>>>>>>>> 
>>>>>>>> Starting next week, yes.
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>>> Having a couple of quieter days, I have been working on a new, little (hopefully) side project that has probably been high up on our radar since the Shalla list has shut down in 2020, or maybe even earlier. The goal of the project is to provide good lists with categories of domain names which are usually used to block access to these domains.
>>>>>>>>>>> 
>>>>>>>>>>> I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists.
>>>>>>>>>>> 
>>>>>>>>>>> How did we get here?
>>>>>>>>>>> 
>>>>>>>>>>> As stated before, the URL filter feature in IPFire has the problem that there are not many good blocklists available any more. There used to be a couple more - most famously the Shalla list - but we are now down to a single list from the University of Toulouse. It is a great list, but it is not always the best fit for all users.
>>>>>>>>>>> 
>>>>>>>>>>> Then there has been talk about whether we could implement more blocking features into IPFire that don’t involve the proxy. Most famously blocking over DNS. The problem here remains a the blocking feature is only as good as the data that is fed into it. Some people have been putting forward a number of lists that were suitable for them, but they would not have replaced the blocking functionality as we know it. Their aim is to provide “one list for everything” but that is not what people usually want. It is targeted at a classic home user and the only separation that is being made is any adult/porn/NSFW content which usually is put into a separate list.
>>>>>>>>>>> 
>>>>>>>>>>> It would have been technically possible to include these lists and let the users decide, but that is not the aim of IPFire. We want to do the job for the user so that their job is getting easier. Including obscure lists that don’t have a clear outline of what they actually want to block (“bad content” is not a category) and passing the burden of figuring out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with cream on top is really not going to work. It is all confusing and will lead to a bad user experience.
>>>>>>>>>>> 
>>>>>>>>>>> An even bigger problem that is however completely impossible to solve is bad licensing of these lists. A user has asked the publisher of the HaGeZi list whether they could be included in IPFire and under what terms. The response was that the list is available under the terms of the GNU General Public License v3, but that does not seem to be true. The list contains data from various sources. Many of them are licensed under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public agreement that this data may be redistributed, there is a huge legal issue here. We would expose our users to potential copyright infringement which we cannot do under any circumstances. Furthermore many lists are available under a non-commercial license which excludes them from being used in any kind of business. Plenty of IPFire systems are running in businesses, if not even the vast majority.
>>>>>>>>>>> 
>>>>>>>>>>> In short, these lists are completely unusable for us. Apart from HaGeZi, I consider OISD to have the same problem.
>>>>>>>>>>> 
>>>>>>>>>>> Enough about all the things that are bad. Let’s talk about the new, good things:
>>>>>>>>>>> 
>>>>>>>>>>> Many blacklists on the internet are an amalgamation of other lists. These lists vary in quality with some of them being not that good and without a clear focus and others being excellent data. Since we don’t have the man power to start from scratch, I felt that we can copy the concept that HaGeZi and OISD have started and simply create a new list that is based on other lists at the beginning to have a good starting point. That way, we have much better control over what is going on these lists and we can shape and mould them as we need them. Most importantly, we don’t create a single lists, but many lists that have a clear focus and allow users to choose what they want to block and what not.
>>>>>>>>>>> 
>>>>>>>>>>> So the current experimental stage that I am in has these lists:
>>>>>>>>>>> 
>>>>>>>>>>> * Ads
>>>>>>>>>>> * Dating
>>>>>>>>>>> * DoH
>>>>>>>>>>> * Gambling
>>>>>>>>>>> * Malware
>>>>>>>>>>> * Porn
>>>>>>>>>>> * Social
>>>>>>>>>>> * Violence
>>>>>>>>>>> 
>>>>>>>>>>> The categories have been determined by what source lists we have available with good data and are compatible with our chosen license CC BY-SA 4.0. This is the same license that we are using for the IPFire Location database, too.
>>>>>>>>>>> 
>>>>>>>>>>> The main use-cases for any kind of blocking are to comply with legal requirements in networks with children (i.e. schools) to remove any kind of pornographic content, sometimes block social media as well. Gambling and violence are commonly blocked, too. Even more common would be filtering advertising and any malicious content.
>>>>>>>>>>> 
>>>>>>>>>>> The latter is especially difficult because so many source lists throw phishing, spyware, malvertising, tracking and other things into the same bucket. Here this is currently all in the malware list which has therefore become quite large. I am not sure whether this will stay like this in the future or if we will have to make some adjustments, but that is exactly why this is now entering some larger testing.
>>>>>>>>>>> 
>>>>>>>>>>> What has been built so far? In order to put these lists together properly, track any data about where it is coming from, I have built a tool in Python available here:
>>>>>>>>>>> 
>>>>>>>>>>> https://git.ipfire.org/?p=dnsbl.git;a=summary
>>>>>>>>>>> 
>>>>>>>>>>> This tool will automatically update all lists once an hour if there have been any changes and export them in various formats. The exported lists are available for download here:
>>>>>>>>>>> 
>>>>>>>>>>> https://dnsbl.ipfire.org/lists/
>>>>>>>>>> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the custom url works fine.
>>>>>>>>>> 
>>>>>>>>>> However you need to remember not to put the https:// at the front of the url otherwise the WUI page completes without any error messages but leaves an error message in the system logs saying
>>>>>>>>>> 
>>>>>>>>>> URL filter blacklist - ERROR: Not a valid URL filter blacklist
>>>>>>>>>> 
>>>>>>>>>> I found this out the hard way.
>>>>>>>>> 
>>>>>>>>> Oh yes, I forgot that there is a field on the web UI. If that does not accept https:// as a prefix, please file a bug and we will fix it.
>>>>>>>> 
>>>>>>>> I will confirm it and raise a bug.
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> The other thing I noticed is that if you already have the Toulouse University list downloaded and you then change to the ipfire custom url then all the existing Toulouse blocklists stay in the directory on IPFire and so you end up with a huge number of category tick boxes, most of which are the old Toulouse ones, which are still available to select and it is not clear which ones are from Toulouse and which ones from IPFire.
>>>>>>>>> 
>>>>>>>>> Yes, I got the same thing, too. I think this is a bug, too, because otherwise you would have a lot of unused categories lying around that will never be updated. You cannot even tell which ones are from the current list and which ones from the old list.
>>>>>>>>> 
>>>>>>>>> Long-term we could even consider to remove the Univ. Toulouse list entirely and only have our own lists available which would make the problem go away.
>>>>>>>>> 
>>>>>>>>>> I think if the blocklist URL source is changed or a custom url is provided the first step should be to remove the old ones already existing.
>>>>>>>>>> That might be a problem because users can also create their own blocklists and I believe those go into the same directory.
>>>>>>>>> 
>>>>>>>>> Good thought. We of course cannot delete the custom lists.
>>>>>>>>> 
>>>>>>>>>> Without clearing out the old blocklists you end up with a huge number of checkboxes for lists but it is not clear what happens if there is a category that has the same name for the Toulouse list and the IPFire list such as gambling. I will have a look at that and see what happens.
>>>>>>>>>> 
>>>>>>>>>> Not sure what the best approach to this is.
>>>>>>>>> 
>>>>>>>>> I believe it is removing all old content.
>>>>>>>>> 
>>>>>>>>>> Manually deleting all contents of the urlfilter/blacklists/ directory and then selecting the IPFire blocklist url for the custom url I end up with only the 8 categories from the IPFire list.
>>>>>>>>>> 
>>>>>>>>>> I have tested some gambling sites from the IPFire list and the block worked on some. On others the site no longer exists so there is nothing to block or has been changed to an https site and in that case it went straight through. Also if I chose the http version of the link, it was automatically changed to https and went through without being blocked.
>>>>>>>>> 
>>>>>>>>> The entire IPFire infrastructure always requires HTTPS. If you start using HTTP, you will be automatically redirected. It is 2026 and we don’t need to talk HTTP any more :)
>>>>>>>> 
>>>>>>>> Some of the domains in the gambling list (maybe quite a lot) seem to only have an http access. If I tried https it came back with the fact that it couldn't find it.
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> I am glad to hear that the list is actually blocking. It would have been bad if it didn’t. Now we have the big task to check out the “quality” - however that can be determined. I think this is what needs some time…
>>>>>>>>> 
>>>>>>>>> In the meantime I have set up a small page on our website:
>>>>>>>>> 
>>>>>>>>> https://www.ipfire.org/dnsbl
>>>>>>>>> 
>>>>>>>>> I would like to run this as a first-class project inside IPFire like we are doing with IPFire Location. That means that we need to tell people about what we are doing. Hopefully this page is a little start.
>>>>>>>>> 
>>>>>>>>> Initially it has a couple of high-level bullet points about what we are trying to achieve. I don’t think the text is very good, yet, but it is the best I had in that moment. There is then also a list of the lists that we currently offer. For each list, a detailed page will tell you about the license, how many domains are listed, when the last update has been, the sources and even there is a history page that shows all the changes whenever they have happened.
>>>>>>>>> 
>>>>>>>>> Finally there is a section that explains “How To Use?” the list which I would love to extend to include AdGuard Plus and things like that as well as Pi-Hole and whatever else could use the list. In a later step we should go ahead and talk to any projects to include our list(s) into their dropdown so that people can enable them nice and easy.
>>>>>>>>> 
>>>>>>>>> Behind the web page there is an API service that is running on the host that is running the DNSBL. The frontend web app that is running www.ipfire.org <http://www.ipfire.org/> is connecting to that API service to fetch the current lists, any details and so on. That way, we can split the logic and avoid creating a huge monolith of a web app. This also means that page could be down a little as I am still working on the entire thing and will frequently restart it.
>>>>>>>>> 
>>>>>>>>> The API documentation is available here and the API is publicly available: https://api.dnsbl.ipfire.org/docs
>>>>>>>>> 
>>>>>>>>> The website/API allows to file reports for anything that does not seem to be right on any of the lists. I would like to keep it as an open process, however, long-term, this cannot cost us any time. In the current stage, the reports are getting filed and that is about it. I still need to build out some way for admins or moderators (I am not sure what kind of roles I want to have here) to accept or reject those reports.
>>>>>>>>> 
>>>>>>>>> In case of us receiving a domain from a source list, I would rather like to submit a report to upstream for them to de-list. That way, we don’t have any admin to do and we are contributing back to other list. That would be a very good thing to do. We cannot however throw tons of emails at some random upstream projects without co-ordinating this first. By not reporting upstream, we will probably over time create large whitelists and I am not sure if that is a good thing to do.
>>>>>>>>> 
>>>>>>>>> Finally, there is a search box that can be used to find out if a domain is listed on any of the lists.
>>>>>>>>> 
>>>>>>>>>>> If you download and open any of the files, you will see a large header that includes copyright information and lists all sources that have been used to create the individual lists. This way we ensure maximum transparency, comply with the terms of the individual licenses of the source lists and give credit to the people who help us to put together the most perfect list for our users.
>>>>>>>>>>> 
>>>>>>>>>>> I would like this to become a project that is not only being used in IPFire. We can and will be compatible with other solutions like AdGuard, PiHole so that people can use our lists if they would like to even though they are not using IPFire. Hopefully, these users will also feed back to us so that we can improve our lists over time and make them one of the best options out there.
>>>>>>>>>>> 
>>>>>>>>>>> All lists are available as a simple text file that lists the domains. Then there is a hosts file available as well as a DNS zone file and an RPZ file. Each list is individually available to be used in squidGuard and there is a larger tarball available with all lists that can be used in IPFire’s URL Filter. I am planning to add Suricata/Snort signatures whenever I have time to do so. Even though it is not a good idea to filter pornographic content this way, I suppose that catching malware and blocking DoH are good use-cases for an IPS. Time will tell…
>>>>>>>>>>> 
>>>>>>>>>>> As a start, we will make these lists available in IPFire’s URL Filter and collect some feedback about how we are doing. Afterwards, we can see where else we can take this project.
>>>>>>>>>>> 
>>>>>>>>>>> If you want to enable this on your system, simply add the URL to your autoupdate.urls file like here:
>>>>>>>>>>> 
>>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f
>>>>>>>>>> I also tested out adding the IPFire url to autoupdate.urls and that also worked fine for me.
>>>>>>>>> 
>>>>>>>>> Very good. Should we include this already with Core Update 200? I don’t think we would break anything, but we might already gain a couple more people who are helping us to test this all?
>>>>>>>> 
>>>>>>>> I think that would be a good idea.
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> The next step would be to build and test our DNS infrastructure. In the “How To Use?” Section on the pages of the individual lists, you can already see some instructions on how to use the lists as an RPZ. In comparison to other “providers”, I would prefer if people would be using DNS to fetch the lists. This is simply to push out updates in a cheap way for us and also do it very regularly.
>>>>>>>>> 
>>>>>>>>> Initially, clients will pull the entire list using AXFR. There is no way around this as they need to have the data in the first place. After that, clients will only need the changes. As you can see in the history, the lists don’t actually change that often. Sometimes only once a day and therefore downloading the entire list again would be a huge waste of data, both on the client side, but also for us hosting then.
>>>>>>>>> 
>>>>>>>>> Some other providers update their lists “every 10 minutes”, and there won't be any changes whatsoever. We don’t do that. We will only export the lists again when they have actually changed. The timestamps on the files that we offer using HTTPS can be checked by clients so that they won’t re-download the list again if it has not been changed. But using HTTPS still means that we would have to re-download the entire list and not only the changes.
>>>>>>>>> 
>>>>>>>>> Using DNS and IXFR will update the lists by only transferring a few kilobytes and therefore we can have clients check once an hour if a list has actually changed and only send out the raw changes. That way, we will be able to serve millions of clients at very cheap cost and they will always have a very up to date list.
>>>>>>>>> 
>>>>>>>>> As far as I can see any DNS software that supports RPZs supports AXFR/IXFR with exception of Knot Resolver which expects the zone to be downloaded externally. There is a ticket for AXFR/IXFR support (https://gitlab.nic.cz/knot/knot-resolver/-/issues/195).
>>>>>>>>> 
>>>>>>>>> Initially, some of the lists have been *huge* which is why a simple HTTP download is not feasible. The porn list was over 100 MiB. We could have spent thousands on just traffic alone which I don’t have for this kind of project. It would also be unnecessary money being spent. There are simply better solutions out there. But then I built something that basically tests the data that we are receiving from upstream but simply checking if a listed domain still exists. The result was very astonishing to me.
>>>>>>>>> 
>>>>>>>>> So whenever someone adds a domain to the list, we will (eventually, but not immediately) check if we can resolve the domain’s SOA record. If not, we mark the domain as non-active and will no longer include them in the exported data. This brought down the porn list from just under 5 million domains to just 421k. On the sources page (https://www.ipfire.org/dnsbl/lists/porn/sources) I am listing the percentage of dead domains from each of them and the UT1 list has 94% dead domains. Wow.
>>>>>>>>> 
>>>>>>>>> If we cannot resolve the domain, neither can our users. So we would otherwise fill the lists with tons of domains that simply could never be reached. And if they cannot be reached, why would we block them? We would waste bandwidth and a lot of memory on each single client.
>>>>>>>>> 
>>>>>>>>> The other sources have similarly high rations of dead domains. Most of them are in the 50-80% range. Therefore I am happy that we are doing some extra work here to give our users much better data for their filtering.
>>>>>>>> 
>>>>>>>> Removing all dead entries sounds like an excellent step.
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> 
>>>>>>>> Adolf.
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> So, if you like, please go and check out the RPZ blocking with Unbound. Instructions are on the page. I would be happy to hear how this is turning out.
>>>>>>>>> 
>>>>>>>>> Please let me know if there are any more questions, and I would be glad to answer them.
>>>>>>>>> 
>>>>>>>>> Happy New Year,
>>>>>>>>> -Michael
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> Adolf.
>>>>>>>>>>> This email is just a brain dump from me to this list. I would be happy to answer any questions about implementation details, etc. if people are interested. Right now, this email is long enough already…
>>>>>>>>>>> 
>>>>>>>>>>> All the best,
>>>>>>>>>>> -Michael
>>>>>>>>>> 
>>>>>>>>>> -- 
>>>>>>>>>> Sent from my laptop
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> -- 
>>>>>> Sent from my laptop
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 

Re: Let's launch our own blocklists...

[Adam Gibbons]

On 22/01/2026 11:33 am, Michael Tremer wrote:
> 
> Hello everyone,

Hi Michael and Everyone,

> Over the past few weeks I have made significant progress on this all, 
> and I think we're getting close to something the community will be 
> really happy with. I'd love to get feedback from the team before we 
> finalise things.

I've been keeping an eye on the development of IPFire DBL (RIP DNSBL) 
since the beginning and I agree the community should be very happy with 
it.

Having our own trusted, properly maintained blocklists that fit nicely 
into IPFire (and the wider adblock/domain-blocking ecosystem) makes 
IPFire a more complete UTM solution while expanding the IPFire brand.

I think we can make IPFire DBL a trusted alternative to the random lists 
hosted on the internet these days. Even the small things like DNSSEC by 
default, removing dead domains,and being able to track 
additions/removals over time will help build trust.

With the help of the community we can further refine the lists and make 
IPFire DBL something that attracts attention to the project as a whole.

> I have added a couple more lists that I thought interesting and I have 
> added a couple more sources that I considered a good start. Hopefully, 
> we will soon gather some more feedback on how well this is all holding 
> up. My main focus has however been on the technology that will power 
> this project.

I've been using several of the lists for a few weeks now and the number 
of false-positives has been minimal for my use-case. It has even passed 
the dreaded "wife test", so I think this is ready for wider testing.

> One of the bigger challenges was to create Suricata rules from the 
> lists. Initially I tried to create a ton of rules but since our lists 
> are so large, this quickly became too complicated. I have now settled 
> on using a feature that is only available in more recent versions of 
> Suricata (I believe 7 and later), but since we are already on Suricata 
> 8 in IPFire this won't be a problem for us. All domains for each list 
> are basically compiled into one massively large dataset and one single 
> rule is referring to that dataset. This way, we won't have the option 
> to remove any false-positives, but at least Suricata and the GUI won't 
> starve a really bad death when loading millions of rules.

Yes, the IPFire DBL rules in IPS currently block all domains on the list 
without the ability to de-select false-positives. Luckily the reporting 
feature is very easy to use.

As mentioned previously, domain blocking in Suricata isn't intended to 
be the optimal way to handle DNS blocking because connections are 
dropped rather than returning a DNS response such as NXDOMAIN, so it's 
best used as a policy backstop.

For scenarios like schools using the "violence" list this will be 
perfect, though it might make some websites look/act unusual if the 
"advertising" list is used. But every scenario is different.

We see similar effects when people enable the "emerging-dns.rule" 
ruleset where the ".cc" TLD is blocked by default. Quite often because 
of the way it's dropped, it's sometimes difficult to diagnose because of 
the unusual UX. It pops up on the forum occasionally.

> Looking ahead, I would like us to start thinking about the RPZ feature 
> that has been on the wishlist. IPFire DBL has been a bigger piece of 
> work, and I think it's worth having a conversation about 
> sustainability. Resources for this need to be allocated and paid for. 
> Open source is about freedom, not free beer — and to keep building 
> features like this, we will need to explore some funding options. I 
> would be interested to hear any ideas you might have that could work 
> for IPFire.

The community has been seeking a solution like this for some time now. 
It's great to see the start of what it will end up being. With community 
support, both funding and false-positive reporting, this can go a long 
way.

Thank you for the work towards this, Michael.

Regards,
Adam

Re: Let's launch our own blocklists...

[Matthias Fischer]

On 23.01.2026 17:39, Michael Tremer wrote:
> Hello Matthias,

Hi Michael,

> Thank you very much for testing IPFire DBL.

No problem - I have news:

After taking a closer look to the IPS system logs, unfortunately I found
some parsing errors:

'suricata' complains about missing ";".

***SNIP***
...
00:32:40 	suricata: 	[13343] <Info> -- Including configuration file
/var/ipfire/suricata/suricata-used-rulesfiles.yaml.
00:32:40 	suricata: 	[13343] <Error> -- no terminating ";" found
00:32:40 	suricata: 	[13343] <Error> -- error parsing signature "drop
dns any any -> any any (msg:"IPFire DBL [Advertising] Blocked DNS
Query"; dns.query; domain; dataset:isset,ads,type string,load
datasets/ads.txt; classtype:policy-violation; priority:3; sid:983041;
rev:1; reference:url,https://www.ipfire.org/dbl/ads; metadata:dbl
ads.dbl.ipfire.org)" from file /var/lib/suricata/ipfire_dnsbl-ads.rules
at line 72
00:32:40 	suricata: 	[13343] <Error> -- no terminating ";" found
...
***SNAP***

I tried, but didn't find the right place for any missing ";".

Can "anyone" confirm?

Best
Matthias

>> On 23 Jan 2026, at 15:02, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>> 
>> On 22.01.2026 12:33, Michael Tremer wrote:
>>> Hello everyone,
>> 
>> Hi,
>> 
>> short feedback from me:
>> 
>> - I activated both the suricata (IPFire DBL - Domain Blocklist) - and
>> the URLfilter lists from 'dbl.ipfire.org'.
> 
> This is an interesting case. What I didn’t manage to test yet is what happens when Suricata blocks the connection first. If URL Filter sees a domain that is being blocked it will either send you an error page if you are using HTTP, or simply close the connection if it is HTTPS. However, when Suricata comes first in the chain (and it will), it might close the connection because URL Filter has received the request. In the case of HTTPS this does not make any difference because the connection will be closed, but in the HTTP case you won’t see an error page any more and instead have the connection closed, too. You are basically losing the explicit error notification which is a little bit annoying.
> 
> We could have the same when we are doing the same with Unbound and DNS filtering. Potentially we would need to whitelist the local DNS resolver then, but how is Suricata supposed to know that the same categories are activated in both places?
> 
>> - I even took the 'smart-tv' domains from the IFire DBL blacklist and
>> copied/pasted them in my fritzbox filter lists.
> 
> LOL Why not use IPFire to filter this as well?
> 
>> Everything works as expected. Besides, the download of the IPFire
>> DBL-list loads a lot faster than the list from 'Univ. Toulouse'... ;-)
> 
> Yes, we don’t have much traffic on the server, yet.
> 
>> Functionality is good - no false positives or seen problems. Good work -
>> thanks!
> 
> Nice. We need to distinguish a little between what is a technical issue and what is a false-positive/missing domain on the list. However, testing both at the same time is something we will all cope quite well with :)
> 
> -Michael
> 
>> Best
>> Matthias
>> 
>>> Over the past few weeks I have made significant progress on this all, and I think we're getting close to something the community will be really happy with. I'd love to get feedback from the team before we finalise things.
>>> 
>>> So what has happened?
>>> 
>>> First of all, the entire project has been renamed. DNSBL is not entirely what this is. Although the lists can be thrown into DNS, they have much more use outside of it that I thought we should simply go with DBL, short for Domain Blocklist. After all, we are only importing domains. The new home of the project therefore is https://www.ipfire.org/dbl
>>> 
>>> I have added a couple more lists that I thought interesting and I have added a couple more sources that I considered a good start. Hopefully, we will soon gather some more feedback on how well this is all holding up. My main focus has however been on the technology that will power this project.
>>> 
>>> One of the bigger challenges was to create Suricata rules from the lists. Initially I tried to create a ton of rules but since our lists are so large, this quickly became too complicated. I have now settled on using a feature that is only available in more recent versions of Suricata (I believe 7 and later), but since we are already on Suricata 8 in IPFire this won’t be a problem for us. All domains for each list are basically compiled into one massively large dataset and one single rule is referring to that dataset. This way, we won’t have the option to remove any false-positives, but at least Suricata and the GUI won’t starve a really bad death when loading millions of rules.
>>> 
>>> Suricata will now be able to use our rules to block access to any listed domains of each of the categories over DNS, HTTP, TLS or QUIC. Although I don’t expect many users to use Suricata to block porn or other things, this is a great backstop to enforce any policy like that. For example, if there is a user on the network who is trying to circumvent the DNS server that might filter out certain domains, even after getting an IP address resolved through other means, they won’t be able to open a TLS/QUIC connection or send a HTTP request to all blocked domains. Some people have said they were interested in blocking DNS-over-HTTPS and this is a perfect way to do this and actually be sure that any server that is being blocked on the list will actually be completely inaccessible.
>>> 
>>> Those Suricata rules are already available for testing in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=9eb8751487d23dd354a105c28bdbbb0398fe6e85
>>> 
>>> I have chosen various severities for the lists. If someone was to block advertising using DBL, this is fine, but not a very severe alert. If someone chooses to block malware and there is a system on the network trying to access those domains, this is an alert worth being investigated by an admin. Our new Suricata Reporter will show those violations in different colours based on the severity which helps to identify the right alerts to further investigate.
>>> 
>>> Formerly I have asked you to test the lists using URL Filter. Those rules are now available as well in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=db160694279a4b10378447f775dd536fdfcfb02a
>>> 
>>> I talked about a method to remove any dead domains from any sources which is a great way to keep our lists smaller. The pure size of them is a problem in so many ways. That check was however a little bit too ambitious and I had to make it a little bit less eager. Basically if we are in doubt, we need to still list the domain because it might be resolvable by a user.
>>> 
>>>  https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=bb5b6e33b731501d45dea293505f7d42a61d5ce7
>>> 
>>> So how else could we make the lists smaller without losing any actual data? Since we sometimes list a whole TLD (e.g. .xxx or .porn), there is very little point in listing any domains of this TLD. They will always be caught anyways. So I built a check that marks all domains that don’t need to be included on the exported lists because they will never be needed and was able to shrink the size of the lists by a lot again.
>>> 
>>> The website does not show this data, but the API returns the number of “subsumed” domains (I didn’t have a better name):
>>> 
>>>  curl https://api.dbl.ipfire.org/lists | jq .
>>> 
>>> The number shown would normally be added to the total number of domains and usually cuts the size of the list by 50-200%.
>>> 
>>> Those stats will now also be stored in a history table so that we will be able to track growth of all lists.
>>> 
>>> Furthermore, the application will now send email notifications for any incoming reports. This way, we will be able to stay in close touch with the reporters and keep them up to date on their submissions as well as inform moderators that there is something to have a look at.
>>> 
>>> The search has been refactored as well, so that we can show clearly whether something is blocked or not at one glance: https://www.ipfire.org/dbl/search?q=github.com. There is detailed information available on all domains and what happened to them. In case of GitHub.com, this seems to be blocked and unblocked by someone all of the time and we can see a clear audit trail of that: https://www.ipfire.org/dbl/lists/malware/domains/github.com
>>> 
>>> On the DNS front, I have added some metadata to the zones so that people can programmatically request some data, like when it has been last updated (in a human-friendly timestamp and not only the serial), license, description and so on:
>>> 
>>>  # dig +short ANY _info.ads.dbl.ipfire.org @primary.dbl.ipfire.org
>>>  "total-domains=42226"
>>>  "license=CC BY-SA 4.0"
>>>  "updated-at=2026-01-20T22:17:02.409933+00:00"
>>>  "description=Blocks domains used for ads, tracking, and ad delivery”
>>> 
>>> Now, I would like to hear more feedback from you. I know we've all been stretched thin lately, so I especially appreciate anyone who has time to review and provide input. Ideas, just say if you like it or not. Where this could go in the future?
>>> 
>>> Looking ahead, I would like us to start thinking about the RPZ feature that has been on the wishlist. IPFire DBL has been a bigger piece of work, and I think it's worth having a conversation about sustainability. Resources for this need to be allocated and paid for. Open source is about freedom, not free beer — and to keep building features like this, we will need to explore some funding options. I would be interested to hear any ideas you might have that could work for IPFire.
>>> 
>>> Please share your thoughts on the mailing list when you can — even a quick 'looks good' or 'I have concerns about X' is valuable. Public discussion helps everyone stay in the loop and contribute.
>>> 
>>> I am aiming to move forward with this in a week's time, so if you have input, now would be a good time to share it.
>>> 
>>> Best,
>>> -Michael
>>> 
>>>> On 6 Jan 2026, at 10:20, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>>> 
>>>> Good Morning Adolf,
>>>> 
>>>> I had a look at this problem yesterday and it seems that parsing the format is becoming a little bit difficult this way. Since this is only affecting very few domains, I have simply whitelisted them all manually and duckduckgo.com <http://duckduckgo.com/> and others should now be easily reachable again.
>>>> 
>>>> Please let me know if you have any more findings.
>>>> 
>>>> All the best,
>>>> -Michael
>>>> 
>>>>> On 5 Jan 2026, at 11:48, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>>>> 
>>>>> Hello Adolf,
>>>>> 
>>>>> This is a good find.
>>>>> 
>>>>> But if duckduckgo.com <http://duckduckgo.com/> is blocked, we will have to have a source somewhere that blocks that domain. Not only a sub-domain of it. Otherwise we have a bug somewhere.
>>>>> 
>>>>> This is most likely as the domain is listed here, but with some stuff afterwards:
>>>>> 
>>>>> https://raw.githubusercontent.com/mtxadmin/ublock/refs/heads/master/hosts/_malware_typo
>>>>> 
>>>>> We strip everything after a # away because we consider it a comment. However, that causes that there is only a line with the domain left which will cause it being listed.
>>>>> 
>>>>> The # sign is used as some special character but at the same time it is being used for comments.
>>>>> 
>>>>> I will fix this and then refresh the list.
>>>>> 
>>>>> -Michael
>>>>> 
>>>>>> On 5 Jan 2026, at 11:31, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>> 
>>>>>> Hi Michael,
>>>>>> 
>>>>>> 
>>>>>> On 05/01/2026 12:11, Adolf Belka wrote:
>>>>>>> Hi Michael,
>>>>>>> 
>>>>>>> I have found that the malware list includes duckduckgo.com
>>>>>>> 
>>>>>> I have checked through the various sources used for the malware list.
>>>>>> 
>>>>>> The ShadowWhisperer (Tracking) list has improving.duckduckgo.com in its list. I suspect that this one is the one causing the problem.
>>>>>> 
>>>>>> The mtxadmin (_malware_typo) list has duckduckgo.com mentioned 3 times but not directly as a domain name - looks more like a reference.
>>>>>> 
>>>>>> Regards,
>>>>>> 
>>>>>> Adolf.
>>>>>> 
>>>>>> 
>>>>>>> Regards,
>>>>>>> Adolf.
>>>>>>> 
>>>>>>> 
>>>>>>> On 02/01/2026 14:02, Adolf Belka wrote:
>>>>>>>> Hi,
>>>>>>>> 
>>>>>>>> On 02/01/2026 12:09, Michael Tremer wrote:
>>>>>>>>> Hello,
>>>>>>>>> 
>>>>>>>>>> On 30 Dec 2025, at 14:05, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi Michael,
>>>>>>>>>> 
>>>>>>>>>> On 29/12/2025 13:05, Michael Tremer wrote:
>>>>>>>>>>> Hello everyone,
>>>>>>>>>>> 
>>>>>>>>>>> I hope everyone had a great Christmas and a couple of quiet days to relax from all the stress that was the year 2025.
>>>>>>>>>> Still relaxing.
>>>>>>>>> 
>>>>>>>>> Very good, so let’s have a strong start into 2026 now!
>>>>>>>> 
>>>>>>>> Starting next week, yes.
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>>> Having a couple of quieter days, I have been working on a new, little (hopefully) side project that has probably been high up on our radar since the Shalla list has shut down in 2020, or maybe even earlier. The goal of the project is to provide good lists with categories of domain names which are usually used to block access to these domains.
>>>>>>>>>>> 
>>>>>>>>>>> I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists.
>>>>>>>>>>> 
>>>>>>>>>>> How did we get here?
>>>>>>>>>>> 
>>>>>>>>>>> As stated before, the URL filter feature in IPFire has the problem that there are not many good blocklists available any more. There used to be a couple more - most famously the Shalla list - but we are now down to a single list from the University of Toulouse. It is a great list, but it is not always the best fit for all users.
>>>>>>>>>>> 
>>>>>>>>>>> Then there has been talk about whether we could implement more blocking features into IPFire that don’t involve the proxy. Most famously blocking over DNS. The problem here remains a the blocking feature is only as good as the data that is fed into it. Some people have been putting forward a number of lists that were suitable for them, but they would not have replaced the blocking functionality as we know it. Their aim is to provide “one list for everything” but that is not what people usually want. It is targeted at a classic home user and the only separation that is being made is any adult/porn/NSFW content which usually is put into a separate list.
>>>>>>>>>>> 
>>>>>>>>>>> It would have been technically possible to include these lists and let the users decide, but that is not the aim of IPFire. We want to do the job for the user so that their job is getting easier. Including obscure lists that don’t have a clear outline of what they actually want to block (“bad content” is not a category) and passing the burden of figuring out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with cream on top is really not going to work. It is all confusing and will lead to a bad user experience.
>>>>>>>>>>> 
>>>>>>>>>>> An even bigger problem that is however completely impossible to solve is bad licensing of these lists. A user has asked the publisher of the HaGeZi list whether they could be included in IPFire and under what terms. The response was that the list is available under the terms of the GNU General Public License v3, but that does not seem to be true. The list contains data from various sources. Many of them are licensed under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public agreement that this data may be redistributed, there is a huge legal issue here. We would expose our users to potential copyright infringement which we cannot do under any circumstances. Furthermore many lists are available under a non-commercial license which excludes them from being used in any kind of business. Plenty of IPFire systems are running in businesses, if not even the vast majority.
>>>>>>>>>>> 
>>>>>>>>>>> In short, these lists are completely unusable for us. Apart from HaGeZi, I consider OISD to have the same problem.
>>>>>>>>>>> 
>>>>>>>>>>> Enough about all the things that are bad. Let’s talk about the new, good things:
>>>>>>>>>>> 
>>>>>>>>>>> Many blacklists on the internet are an amalgamation of other lists. These lists vary in quality with some of them being not that good and without a clear focus and others being excellent data. Since we don’t have the man power to start from scratch, I felt that we can copy the concept that HaGeZi and OISD have started and simply create a new list that is based on other lists at the beginning to have a good starting point. That way, we have much better control over what is going on these lists and we can shape and mould them as we need them. Most importantly, we don’t create a single lists, but many lists that have a clear focus and allow users to choose what they want to block and what not.
>>>>>>>>>>> 
>>>>>>>>>>> So the current experimental stage that I am in has these lists:
>>>>>>>>>>> 
>>>>>>>>>>> * Ads
>>>>>>>>>>> * Dating
>>>>>>>>>>> * DoH
>>>>>>>>>>> * Gambling
>>>>>>>>>>> * Malware
>>>>>>>>>>> * Porn
>>>>>>>>>>> * Social
>>>>>>>>>>> * Violence
>>>>>>>>>>> 
>>>>>>>>>>> The categories have been determined by what source lists we have available with good data and are compatible with our chosen license CC BY-SA 4.0. This is the same license that we are using for the IPFire Location database, too.
>>>>>>>>>>> 
>>>>>>>>>>> The main use-cases for any kind of blocking are to comply with legal requirements in networks with children (i.e. schools) to remove any kind of pornographic content, sometimes block social media as well. Gambling and violence are commonly blocked, too. Even more common would be filtering advertising and any malicious content.
>>>>>>>>>>> 
>>>>>>>>>>> The latter is especially difficult because so many source lists throw phishing, spyware, malvertising, tracking and other things into the same bucket. Here this is currently all in the malware list which has therefore become quite large. I am not sure whether this will stay like this in the future or if we will have to make some adjustments, but that is exactly why this is now entering some larger testing.
>>>>>>>>>>> 
>>>>>>>>>>> What has been built so far? In order to put these lists together properly, track any data about where it is coming from, I have built a tool in Python available here:
>>>>>>>>>>> 
>>>>>>>>>>> https://git.ipfire.org/?p=dnsbl.git;a=summary
>>>>>>>>>>> 
>>>>>>>>>>> This tool will automatically update all lists once an hour if there have been any changes and export them in various formats. The exported lists are available for download here:
>>>>>>>>>>> 
>>>>>>>>>>> https://dnsbl.ipfire.org/lists/
>>>>>>>>>> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the custom url works fine.
>>>>>>>>>> 
>>>>>>>>>> However you need to remember not to put the https:// at the front of the url otherwise the WUI page completes without any error messages but leaves an error message in the system logs saying
>>>>>>>>>> 
>>>>>>>>>> URL filter blacklist - ERROR: Not a valid URL filter blacklist
>>>>>>>>>> 
>>>>>>>>>> I found this out the hard way.
>>>>>>>>> 
>>>>>>>>> Oh yes, I forgot that there is a field on the web UI. If that does not accept https:// as a prefix, please file a bug and we will fix it.
>>>>>>>> 
>>>>>>>> I will confirm it and raise a bug.
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> The other thing I noticed is that if you already have the Toulouse University list downloaded and you then change to the ipfire custom url then all the existing Toulouse blocklists stay in the directory on IPFire and so you end up with a huge number of category tick boxes, most of which are the old Toulouse ones, which are still available to select and it is not clear which ones are from Toulouse and which ones from IPFire.
>>>>>>>>> 
>>>>>>>>> Yes, I got the same thing, too. I think this is a bug, too, because otherwise you would have a lot of unused categories lying around that will never be updated. You cannot even tell which ones are from the current list and which ones from the old list.
>>>>>>>>> 
>>>>>>>>> Long-term we could even consider to remove the Univ. Toulouse list entirely and only have our own lists available which would make the problem go away.
>>>>>>>>> 
>>>>>>>>>> I think if the blocklist URL source is changed or a custom url is provided the first step should be to remove the old ones already existing.
>>>>>>>>>> That might be a problem because users can also create their own blocklists and I believe those go into the same directory.
>>>>>>>>> 
>>>>>>>>> Good thought. We of course cannot delete the custom lists.
>>>>>>>>> 
>>>>>>>>>> Without clearing out the old blocklists you end up with a huge number of checkboxes for lists but it is not clear what happens if there is a category that has the same name for the Toulouse list and the IPFire list such as gambling. I will have a look at that and see what happens.
>>>>>>>>>> 
>>>>>>>>>> Not sure what the best approach to this is.
>>>>>>>>> 
>>>>>>>>> I believe it is removing all old content.
>>>>>>>>> 
>>>>>>>>>> Manually deleting all contents of the urlfilter/blacklists/ directory and then selecting the IPFire blocklist url for the custom url I end up with only the 8 categories from the IPFire list.
>>>>>>>>>> 
>>>>>>>>>> I have tested some gambling sites from the IPFire list and the block worked on some. On others the site no longer exists so there is nothing to block or has been changed to an https site and in that case it went straight through. Also if I chose the http version of the link, it was automatically changed to https and went through without being blocked.
>>>>>>>>> 
>>>>>>>>> The entire IPFire infrastructure always requires HTTPS. If you start using HTTP, you will be automatically redirected. It is 2026 and we don’t need to talk HTTP any more :)
>>>>>>>> 
>>>>>>>> Some of the domains in the gambling list (maybe quite a lot) seem to only have an http access. If I tried https it came back with the fact that it couldn't find it.
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> I am glad to hear that the list is actually blocking. It would have been bad if it didn’t. Now we have the big task to check out the “quality” - however that can be determined. I think this is what needs some time…
>>>>>>>>> 
>>>>>>>>> In the meantime I have set up a small page on our website:
>>>>>>>>> 
>>>>>>>>> https://www.ipfire.org/dnsbl
>>>>>>>>> 
>>>>>>>>> I would like to run this as a first-class project inside IPFire like we are doing with IPFire Location. That means that we need to tell people about what we are doing. Hopefully this page is a little start.
>>>>>>>>> 
>>>>>>>>> Initially it has a couple of high-level bullet points about what we are trying to achieve. I don’t think the text is very good, yet, but it is the best I had in that moment. There is then also a list of the lists that we currently offer. For each list, a detailed page will tell you about the license, how many domains are listed, when the last update has been, the sources and even there is a history page that shows all the changes whenever they have happened.
>>>>>>>>> 
>>>>>>>>> Finally there is a section that explains “How To Use?” the list which I would love to extend to include AdGuard Plus and things like that as well as Pi-Hole and whatever else could use the list. In a later step we should go ahead and talk to any projects to include our list(s) into their dropdown so that people can enable them nice and easy.
>>>>>>>>> 
>>>>>>>>> Behind the web page there is an API service that is running on the host that is running the DNSBL. The frontend web app that is running www.ipfire.org <http://www.ipfire.org/> is connecting to that API service to fetch the current lists, any details and so on. That way, we can split the logic and avoid creating a huge monolith of a web app. This also means that page could be down a little as I am still working on the entire thing and will frequently restart it.
>>>>>>>>> 
>>>>>>>>> The API documentation is available here and the API is publicly available: https://api.dnsbl.ipfire.org/docs
>>>>>>>>> 
>>>>>>>>> The website/API allows to file reports for anything that does not seem to be right on any of the lists. I would like to keep it as an open process, however, long-term, this cannot cost us any time. In the current stage, the reports are getting filed and that is about it. I still need to build out some way for admins or moderators (I am not sure what kind of roles I want to have here) to accept or reject those reports.
>>>>>>>>> 
>>>>>>>>> In case of us receiving a domain from a source list, I would rather like to submit a report to upstream for them to de-list. That way, we don’t have any admin to do and we are contributing back to other list. That would be a very good thing to do. We cannot however throw tons of emails at some random upstream projects without co-ordinating this first. By not reporting upstream, we will probably over time create large whitelists and I am not sure if that is a good thing to do.
>>>>>>>>> 
>>>>>>>>> Finally, there is a search box that can be used to find out if a domain is listed on any of the lists.
>>>>>>>>> 
>>>>>>>>>>> If you download and open any of the files, you will see a large header that includes copyright information and lists all sources that have been used to create the individual lists. This way we ensure maximum transparency, comply with the terms of the individual licenses of the source lists and give credit to the people who help us to put together the most perfect list for our users.
>>>>>>>>>>> 
>>>>>>>>>>> I would like this to become a project that is not only being used in IPFire. We can and will be compatible with other solutions like AdGuard, PiHole so that people can use our lists if they would like to even though they are not using IPFire. Hopefully, these users will also feed back to us so that we can improve our lists over time and make them one of the best options out there.
>>>>>>>>>>> 
>>>>>>>>>>> All lists are available as a simple text file that lists the domains. Then there is a hosts file available as well as a DNS zone file and an RPZ file. Each list is individually available to be used in squidGuard and there is a larger tarball available with all lists that can be used in IPFire’s URL Filter. I am planning to add Suricata/Snort signatures whenever I have time to do so. Even though it is not a good idea to filter pornographic content this way, I suppose that catching malware and blocking DoH are good use-cases for an IPS. Time will tell…
>>>>>>>>>>> 
>>>>>>>>>>> As a start, we will make these lists available in IPFire’s URL Filter and collect some feedback about how we are doing. Afterwards, we can see where else we can take this project.
>>>>>>>>>>> 
>>>>>>>>>>> If you want to enable this on your system, simply add the URL to your autoupdate.urls file like here:
>>>>>>>>>>> 
>>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f
>>>>>>>>>> I also tested out adding the IPFire url to autoupdate.urls and that also worked fine for me.
>>>>>>>>> 
>>>>>>>>> Very good. Should we include this already with Core Update 200? I don’t think we would break anything, but we might already gain a couple more people who are helping us to test this all?
>>>>>>>> 
>>>>>>>> I think that would be a good idea.
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> The next step would be to build and test our DNS infrastructure. In the “How To Use?” Section on the pages of the individual lists, you can already see some instructions on how to use the lists as an RPZ. In comparison to other “providers”, I would prefer if people would be using DNS to fetch the lists. This is simply to push out updates in a cheap way for us and also do it very regularly.
>>>>>>>>> 
>>>>>>>>> Initially, clients will pull the entire list using AXFR. There is no way around this as they need to have the data in the first place. After that, clients will only need the changes. As you can see in the history, the lists don’t actually change that often. Sometimes only once a day and therefore downloading the entire list again would be a huge waste of data, both on the client side, but also for us hosting then.
>>>>>>>>> 
>>>>>>>>> Some other providers update their lists “every 10 minutes”, and there won't be any changes whatsoever. We don’t do that. We will only export the lists again when they have actually changed. The timestamps on the files that we offer using HTTPS can be checked by clients so that they won’t re-download the list again if it has not been changed. But using HTTPS still means that we would have to re-download the entire list and not only the changes.
>>>>>>>>> 
>>>>>>>>> Using DNS and IXFR will update the lists by only transferring a few kilobytes and therefore we can have clients check once an hour if a list has actually changed and only send out the raw changes. That way, we will be able to serve millions of clients at very cheap cost and they will always have a very up to date list.
>>>>>>>>> 
>>>>>>>>> As far as I can see any DNS software that supports RPZs supports AXFR/IXFR with exception of Knot Resolver which expects the zone to be downloaded externally. There is a ticket for AXFR/IXFR support (https://gitlab.nic.cz/knot/knot-resolver/-/issues/195).
>>>>>>>>> 
>>>>>>>>> Initially, some of the lists have been *huge* which is why a simple HTTP download is not feasible. The porn list was over 100 MiB. We could have spent thousands on just traffic alone which I don’t have for this kind of project. It would also be unnecessary money being spent. There are simply better solutions out there. But then I built something that basically tests the data that we are receiving from upstream but simply checking if a listed domain still exists. The result was very astonishing to me.
>>>>>>>>> 
>>>>>>>>> So whenever someone adds a domain to the list, we will (eventually, but not immediately) check if we can resolve the domain’s SOA record. If not, we mark the domain as non-active and will no longer include them in the exported data. This brought down the porn list from just under 5 million domains to just 421k. On the sources page (https://www.ipfire.org/dnsbl/lists/porn/sources) I am listing the percentage of dead domains from each of them and the UT1 list has 94% dead domains. Wow.
>>>>>>>>> 
>>>>>>>>> If we cannot resolve the domain, neither can our users. So we would otherwise fill the lists with tons of domains that simply could never be reached. And if they cannot be reached, why would we block them? We would waste bandwidth and a lot of memory on each single client.
>>>>>>>>> 
>>>>>>>>> The other sources have similarly high rations of dead domains. Most of them are in the 50-80% range. Therefore I am happy that we are doing some extra work here to give our users much better data for their filtering.
>>>>>>>> 
>>>>>>>> Removing all dead entries sounds like an excellent step.
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> 
>>>>>>>> Adolf.
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> So, if you like, please go and check out the RPZ blocking with Unbound. Instructions are on the page. I would be happy to hear how this is turning out.
>>>>>>>>> 
>>>>>>>>> Please let me know if there are any more questions, and I would be glad to answer them.
>>>>>>>>> 
>>>>>>>>> Happy New Year,
>>>>>>>>> -Michael
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> Adolf.
>>>>>>>>>>> This email is just a brain dump from me to this list. I would be happy to answer any questions about implementation details, etc. if people are interested. Right now, this email is long enough already…
>>>>>>>>>>> 
>>>>>>>>>>> All the best,
>>>>>>>>>>> -Michael
>>>>>>>>>> 
>>>>>>>>>> -- 
>>>>>>>>>> Sent from my laptop
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> -- 
>>>>>> Sent from my laptop
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 

Re: Let's launch our own blocklists...

[Michael Tremer]

Hello Matthias,

Nice catch!

I fixed it here and added the missing “;”:

  https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=775561e322ceed43e255e5547bd76047b9f8a40b

If you go to the provider settings there is a button to force a ruleset update which should give you the fixed version. Please let me know if this works.

Best,
-Michael

> On 24 Jan 2026, at 23:41, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
> 
> On 23.01.2026 17:39, Michael Tremer wrote:
>> Hello Matthias,
> 
> Hi Michael,
> 
>> Thank you very much for testing IPFire DBL.
> 
> No problem - I have news:
> 
> After taking a closer look to the IPS system logs, unfortunately I found
> some parsing errors:
> 
> 'suricata' complains about missing ";".
> 
> ***SNIP***
> ...
> 00:32:40 suricata: [13343] <Info> -- Including configuration file
> /var/ipfire/suricata/suricata-used-rulesfiles.yaml.
> 00:32:40 suricata: [13343] <Error> -- no terminating ";" found
> 00:32:40 suricata: [13343] <Error> -- error parsing signature "drop
> dns any any -> any any (msg:"IPFire DBL [Advertising] Blocked DNS
> Query"; dns.query; domain; dataset:isset,ads,type string,load
> datasets/ads.txt; classtype:policy-violation; priority:3; sid:983041;
> rev:1; reference:url,https://www.ipfire.org/dbl/ads; metadata:dbl
> ads.dbl.ipfire.org)" from file /var/lib/suricata/ipfire_dnsbl-ads.rules
> at line 72
> 00:32:40 suricata: [13343] <Error> -- no terminating ";" found
> ...
> ***SNAP***
> 
> I tried, but didn't find the right place for any missing ";".
> 
> Can "anyone" confirm?
> 
> Best
> Matthias
> 
>>> On 23 Jan 2026, at 15:02, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>>> 
>>> On 22.01.2026 12:33, Michael Tremer wrote:
>>>> Hello everyone,
>>> 
>>> Hi,
>>> 
>>> short feedback from me:
>>> 
>>> - I activated both the suricata (IPFire DBL - Domain Blocklist) - and
>>> the URLfilter lists from 'dbl.ipfire.org'.
>> 
>> This is an interesting case. What I didn’t manage to test yet is what happens when Suricata blocks the connection first. If URL Filter sees a domain that is being blocked it will either send you an error page if you are using HTTP, or simply close the connection if it is HTTPS. However, when Suricata comes first in the chain (and it will), it might close the connection because URL Filter has received the request. In the case of HTTPS this does not make any difference because the connection will be closed, but in the HTTP case you won’t see an error page any more and instead have the connection closed, too. You are basically losing the explicit error notification which is a little bit annoying.
>> 
>> We could have the same when we are doing the same with Unbound and DNS filtering. Potentially we would need to whitelist the local DNS resolver then, but how is Suricata supposed to know that the same categories are activated in both places?
>> 
>>> - I even took the 'smart-tv' domains from the IFire DBL blacklist and
>>> copied/pasted them in my fritzbox filter lists.
>> 
>> LOL Why not use IPFire to filter this as well?
>> 
>>> Everything works as expected. Besides, the download of the IPFire
>>> DBL-list loads a lot faster than the list from 'Univ. Toulouse'... ;-)
>> 
>> Yes, we don’t have much traffic on the server, yet.
>> 
>>> Functionality is good - no false positives or seen problems. Good work -
>>> thanks!
>> 
>> Nice. We need to distinguish a little between what is a technical issue and what is a false-positive/missing domain on the list. However, testing both at the same time is something we will all cope quite well with :)
>> 
>> -Michael
>> 
>>> Best
>>> Matthias
>>> 
>>>> Over the past few weeks I have made significant progress on this all, and I think we're getting close to something the community will be really happy with. I'd love to get feedback from the team before we finalise things.
>>>> 
>>>> So what has happened?
>>>> 
>>>> First of all, the entire project has been renamed. DNSBL is not entirely what this is. Although the lists can be thrown into DNS, they have much more use outside of it that I thought we should simply go with DBL, short for Domain Blocklist. After all, we are only importing domains. The new home of the project therefore is https://www.ipfire.org/dbl
>>>> 
>>>> I have added a couple more lists that I thought interesting and I have added a couple more sources that I considered a good start. Hopefully, we will soon gather some more feedback on how well this is all holding up. My main focus has however been on the technology that will power this project.
>>>> 
>>>> One of the bigger challenges was to create Suricata rules from the lists. Initially I tried to create a ton of rules but since our lists are so large, this quickly became too complicated. I have now settled on using a feature that is only available in more recent versions of Suricata (I believe 7 and later), but since we are already on Suricata 8 in IPFire this won’t be a problem for us. All domains for each list are basically compiled into one massively large dataset and one single rule is referring to that dataset. This way, we won’t have the option to remove any false-positives, but at least Suricata and the GUI won’t starve a really bad death when loading millions of rules.
>>>> 
>>>> Suricata will now be able to use our rules to block access to any listed domains of each of the categories over DNS, HTTP, TLS or QUIC. Although I don’t expect many users to use Suricata to block porn or other things, this is a great backstop to enforce any policy like that. For example, if there is a user on the network who is trying to circumvent the DNS server that might filter out certain domains, even after getting an IP address resolved through other means, they won’t be able to open a TLS/QUIC connection or send a HTTP request to all blocked domains. Some people have said they were interested in blocking DNS-over-HTTPS and this is a perfect way to do this and actually be sure that any server that is being blocked on the list will actually be completely inaccessible.
>>>> 
>>>> Those Suricata rules are already available for testing in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=9eb8751487d23dd354a105c28bdbbb0398fe6e85
>>>> 
>>>> I have chosen various severities for the lists. If someone was to block advertising using DBL, this is fine, but not a very severe alert. If someone chooses to block malware and there is a system on the network trying to access those domains, this is an alert worth being investigated by an admin. Our new Suricata Reporter will show those violations in different colours based on the severity which helps to identify the right alerts to further investigate.
>>>> 
>>>> Formerly I have asked you to test the lists using URL Filter. Those rules are now available as well in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=db160694279a4b10378447f775dd536fdfcfb02a
>>>> 
>>>> I talked about a method to remove any dead domains from any sources which is a great way to keep our lists smaller. The pure size of them is a problem in so many ways. That check was however a little bit too ambitious and I had to make it a little bit less eager. Basically if we are in doubt, we need to still list the domain because it might be resolvable by a user.
>>>> 
>>>> https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=bb5b6e33b731501d45dea293505f7d42a61d5ce7
>>>> 
>>>> So how else could we make the lists smaller without losing any actual data? Since we sometimes list a whole TLD (e.g. .xxx or .porn), there is very little point in listing any domains of this TLD. They will always be caught anyways. So I built a check that marks all domains that don’t need to be included on the exported lists because they will never be needed and was able to shrink the size of the lists by a lot again.
>>>> 
>>>> The website does not show this data, but the API returns the number of “subsumed” domains (I didn’t have a better name):
>>>> 
>>>> curl https://api.dbl.ipfire.org/lists | jq .
>>>> 
>>>> The number shown would normally be added to the total number of domains and usually cuts the size of the list by 50-200%.
>>>> 
>>>> Those stats will now also be stored in a history table so that we will be able to track growth of all lists.
>>>> 
>>>> Furthermore, the application will now send email notifications for any incoming reports. This way, we will be able to stay in close touch with the reporters and keep them up to date on their submissions as well as inform moderators that there is something to have a look at.
>>>> 
>>>> The search has been refactored as well, so that we can show clearly whether something is blocked or not at one glance: https://www.ipfire.org/dbl/search?q=github.com. There is detailed information available on all domains and what happened to them. In case of GitHub.com, this seems to be blocked and unblocked by someone all of the time and we can see a clear audit trail of that: https://www.ipfire.org/dbl/lists/malware/domains/github.com
>>>> 
>>>> On the DNS front, I have added some metadata to the zones so that people can programmatically request some data, like when it has been last updated (in a human-friendly timestamp and not only the serial), license, description and so on:
>>>> 
>>>> # dig +short ANY _info.ads.dbl.ipfire.org @primary.dbl.ipfire.org
>>>> "total-domains=42226"
>>>> "license=CC BY-SA 4.0"
>>>> "updated-at=2026-01-20T22:17:02.409933+00:00"
>>>> "description=Blocks domains used for ads, tracking, and ad delivery”
>>>> 
>>>> Now, I would like to hear more feedback from you. I know we've all been stretched thin lately, so I especially appreciate anyone who has time to review and provide input. Ideas, just say if you like it or not. Where this could go in the future?
>>>> 
>>>> Looking ahead, I would like us to start thinking about the RPZ feature that has been on the wishlist. IPFire DBL has been a bigger piece of work, and I think it's worth having a conversation about sustainability. Resources for this need to be allocated and paid for. Open source is about freedom, not free beer — and to keep building features like this, we will need to explore some funding options. I would be interested to hear any ideas you might have that could work for IPFire.
>>>> 
>>>> Please share your thoughts on the mailing list when you can — even a quick 'looks good' or 'I have concerns about X' is valuable. Public discussion helps everyone stay in the loop and contribute.
>>>> 
>>>> I am aiming to move forward with this in a week's time, so if you have input, now would be a good time to share it.
>>>> 
>>>> Best,
>>>> -Michael
>>>> 
>>>>> On 6 Jan 2026, at 10:20, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>>>> 
>>>>> Good Morning Adolf,
>>>>> 
>>>>> I had a look at this problem yesterday and it seems that parsing the format is becoming a little bit difficult this way. Since this is only affecting very few domains, I have simply whitelisted them all manually and duckduckgo.com <http://duckduckgo.com/> and others should now be easily reachable again.
>>>>> 
>>>>> Please let me know if you have any more findings.
>>>>> 
>>>>> All the best,
>>>>> -Michael
>>>>> 
>>>>>> On 5 Jan 2026, at 11:48, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>>>>> 
>>>>>> Hello Adolf,
>>>>>> 
>>>>>> This is a good find.
>>>>>> 
>>>>>> But if duckduckgo.com <http://duckduckgo.com/> is blocked, we will have to have a source somewhere that blocks that domain. Not only a sub-domain of it. Otherwise we have a bug somewhere.
>>>>>> 
>>>>>> This is most likely as the domain is listed here, but with some stuff afterwards:
>>>>>> 
>>>>>> https://raw.githubusercontent.com/mtxadmin/ublock/refs/heads/master/hosts/_malware_typo
>>>>>> 
>>>>>> We strip everything after a # away because we consider it a comment. However, that causes that there is only a line with the domain left which will cause it being listed.
>>>>>> 
>>>>>> The # sign is used as some special character but at the same time it is being used for comments.
>>>>>> 
>>>>>> I will fix this and then refresh the list.
>>>>>> 
>>>>>> -Michael
>>>>>> 
>>>>>>> On 5 Jan 2026, at 11:31, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>> 
>>>>>>> Hi Michael,
>>>>>>> 
>>>>>>> 
>>>>>>> On 05/01/2026 12:11, Adolf Belka wrote:
>>>>>>>> Hi Michael,
>>>>>>>> 
>>>>>>>> I have found that the malware list includes duckduckgo.com
>>>>>>>> 
>>>>>>> I have checked through the various sources used for the malware list.
>>>>>>> 
>>>>>>> The ShadowWhisperer (Tracking) list has improving.duckduckgo.com in its list. I suspect that this one is the one causing the problem.
>>>>>>> 
>>>>>>> The mtxadmin (_malware_typo) list has duckduckgo.com mentioned 3 times but not directly as a domain name - looks more like a reference.
>>>>>>> 
>>>>>>> Regards,
>>>>>>> 
>>>>>>> Adolf.
>>>>>>> 
>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> Adolf.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 02/01/2026 14:02, Adolf Belka wrote:
>>>>>>>>> Hi,
>>>>>>>>> 
>>>>>>>>> On 02/01/2026 12:09, Michael Tremer wrote:
>>>>>>>>>> Hello,
>>>>>>>>>> 
>>>>>>>>>>> On 30 Dec 2025, at 14:05, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Hi Michael,
>>>>>>>>>>> 
>>>>>>>>>>> On 29/12/2025 13:05, Michael Tremer wrote:
>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>> 
>>>>>>>>>>>> I hope everyone had a great Christmas and a couple of quiet days to relax from all the stress that was the year 2025.
>>>>>>>>>>> Still relaxing.
>>>>>>>>>> 
>>>>>>>>>> Very good, so let’s have a strong start into 2026 now!
>>>>>>>>> 
>>>>>>>>> Starting next week, yes.
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>>> Having a couple of quieter days, I have been working on a new, little (hopefully) side project that has probably been high up on our radar since the Shalla list has shut down in 2020, or maybe even earlier. The goal of the project is to provide good lists with categories of domain names which are usually used to block access to these domains.
>>>>>>>>>>>> 
>>>>>>>>>>>> I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists.
>>>>>>>>>>>> 
>>>>>>>>>>>> How did we get here?
>>>>>>>>>>>> 
>>>>>>>>>>>> As stated before, the URL filter feature in IPFire has the problem that there are not many good blocklists available any more. There used to be a couple more - most famously the Shalla list - but we are now down to a single list from the University of Toulouse. It is a great list, but it is not always the best fit for all users.
>>>>>>>>>>>> 
>>>>>>>>>>>> Then there has been talk about whether we could implement more blocking features into IPFire that don’t involve the proxy. Most famously blocking over DNS. The problem here remains a the blocking feature is only as good as the data that is fed into it. Some people have been putting forward a number of lists that were suitable for them, but they would not have replaced the blocking functionality as we know it. Their aim is to provide “one list for everything” but that is not what people usually want. It is targeted at a classic home user and the only separation that is being made is any adult/porn/NSFW content which usually is put into a separate list.
>>>>>>>>>>>> 
>>>>>>>>>>>> It would have been technically possible to include these lists and let the users decide, but that is not the aim of IPFire. We want to do the job for the user so that their job is getting easier. Including obscure lists that don’t have a clear outline of what they actually want to block (“bad content” is not a category) and passing the burden of figuring out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with cream on top is really not going to work. It is all confusing and will lead to a bad user experience.
>>>>>>>>>>>> 
>>>>>>>>>>>> An even bigger problem that is however completely impossible to solve is bad licensing of these lists. A user has asked the publisher of the HaGeZi list whether they could be included in IPFire and under what terms. The response was that the list is available under the terms of the GNU General Public License v3, but that does not seem to be true. The list contains data from various sources. Many of them are licensed under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public agreement that this data may be redistributed, there is a huge legal issue here. We would expose our users to potential copyright infringement which we cannot do under any circumstances. Furthermore many lists are available under a non-commercial license which excludes them from being used in any kind of business. Plenty of IPFire systems are running in businesses, if not even the vast majority.
>>>>>>>>>>>> 
>>>>>>>>>>>> In short, these lists are completely unusable for us. Apart from HaGeZi, I consider OISD to have the same problem.
>>>>>>>>>>>> 
>>>>>>>>>>>> Enough about all the things that are bad. Let’s talk about the new, good things:
>>>>>>>>>>>> 
>>>>>>>>>>>> Many blacklists on the internet are an amalgamation of other lists. These lists vary in quality with some of them being not that good and without a clear focus and others being excellent data. Since we don’t have the man power to start from scratch, I felt that we can copy the concept that HaGeZi and OISD have started and simply create a new list that is based on other lists at the beginning to have a good starting point. That way, we have much better control over what is going on these lists and we can shape and mould them as we need them. Most importantly, we don’t create a single lists, but many lists that have a clear focus and allow users to choose what they want to block and what not.
>>>>>>>>>>>> 
>>>>>>>>>>>> So the current experimental stage that I am in has these lists:
>>>>>>>>>>>> 
>>>>>>>>>>>> * Ads
>>>>>>>>>>>> * Dating
>>>>>>>>>>>> * DoH
>>>>>>>>>>>> * Gambling
>>>>>>>>>>>> * Malware
>>>>>>>>>>>> * Porn
>>>>>>>>>>>> * Social
>>>>>>>>>>>> * Violence
>>>>>>>>>>>> 
>>>>>>>>>>>> The categories have been determined by what source lists we have available with good data and are compatible with our chosen license CC BY-SA 4.0. This is the same license that we are using for the IPFire Location database, too.
>>>>>>>>>>>> 
>>>>>>>>>>>> The main use-cases for any kind of blocking are to comply with legal requirements in networks with children (i.e. schools) to remove any kind of pornographic content, sometimes block social media as well. Gambling and violence are commonly blocked, too. Even more common would be filtering advertising and any malicious content.
>>>>>>>>>>>> 
>>>>>>>>>>>> The latter is especially difficult because so many source lists throw phishing, spyware, malvertising, tracking and other things into the same bucket. Here this is currently all in the malware list which has therefore become quite large. I am not sure whether this will stay like this in the future or if we will have to make some adjustments, but that is exactly why this is now entering some larger testing.
>>>>>>>>>>>> 
>>>>>>>>>>>> What has been built so far? In order to put these lists together properly, track any data about where it is coming from, I have built a tool in Python available here:
>>>>>>>>>>>> 
>>>>>>>>>>>> https://git.ipfire.org/?p=dnsbl.git;a=summary
>>>>>>>>>>>> 
>>>>>>>>>>>> This tool will automatically update all lists once an hour if there have been any changes and export them in various formats. The exported lists are available for download here:
>>>>>>>>>>>> 
>>>>>>>>>>>> https://dnsbl.ipfire.org/lists/
>>>>>>>>>>> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the custom url works fine.
>>>>>>>>>>> 
>>>>>>>>>>> However you need to remember not to put the https:// at the front of the url otherwise the WUI page completes without any error messages but leaves an error message in the system logs saying
>>>>>>>>>>> 
>>>>>>>>>>> URL filter blacklist - ERROR: Not a valid URL filter blacklist
>>>>>>>>>>> 
>>>>>>>>>>> I found this out the hard way.
>>>>>>>>>> 
>>>>>>>>>> Oh yes, I forgot that there is a field on the web UI. If that does not accept https:// as a prefix, please file a bug and we will fix it.
>>>>>>>>> 
>>>>>>>>> I will confirm it and raise a bug.
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> The other thing I noticed is that if you already have the Toulouse University list downloaded and you then change to the ipfire custom url then all the existing Toulouse blocklists stay in the directory on IPFire and so you end up with a huge number of category tick boxes, most of which are the old Toulouse ones, which are still available to select and it is not clear which ones are from Toulouse and which ones from IPFire.
>>>>>>>>>> 
>>>>>>>>>> Yes, I got the same thing, too. I think this is a bug, too, because otherwise you would have a lot of unused categories lying around that will never be updated. You cannot even tell which ones are from the current list and which ones from the old list.
>>>>>>>>>> 
>>>>>>>>>> Long-term we could even consider to remove the Univ. Toulouse list entirely and only have our own lists available which would make the problem go away.
>>>>>>>>>> 
>>>>>>>>>>> I think if the blocklist URL source is changed or a custom url is provided the first step should be to remove the old ones already existing.
>>>>>>>>>>> That might be a problem because users can also create their own blocklists and I believe those go into the same directory.
>>>>>>>>>> 
>>>>>>>>>> Good thought. We of course cannot delete the custom lists.
>>>>>>>>>> 
>>>>>>>>>>> Without clearing out the old blocklists you end up with a huge number of checkboxes for lists but it is not clear what happens if there is a category that has the same name for the Toulouse list and the IPFire list such as gambling. I will have a look at that and see what happens.
>>>>>>>>>>> 
>>>>>>>>>>> Not sure what the best approach to this is.
>>>>>>>>>> 
>>>>>>>>>> I believe it is removing all old content.
>>>>>>>>>> 
>>>>>>>>>>> Manually deleting all contents of the urlfilter/blacklists/ directory and then selecting the IPFire blocklist url for the custom url I end up with only the 8 categories from the IPFire list.
>>>>>>>>>>> 
>>>>>>>>>>> I have tested some gambling sites from the IPFire list and the block worked on some. On others the site no longer exists so there is nothing to block or has been changed to an https site and in that case it went straight through. Also if I chose the http version of the link, it was automatically changed to https and went through without being blocked.
>>>>>>>>>> 
>>>>>>>>>> The entire IPFire infrastructure always requires HTTPS. If you start using HTTP, you will be automatically redirected. It is 2026 and we don’t need to talk HTTP any more :)
>>>>>>>>> 
>>>>>>>>> Some of the domains in the gambling list (maybe quite a lot) seem to only have an http access. If I tried https it came back with the fact that it couldn't find it.
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> I am glad to hear that the list is actually blocking. It would have been bad if it didn’t. Now we have the big task to check out the “quality” - however that can be determined. I think this is what needs some time…
>>>>>>>>>> 
>>>>>>>>>> In the meantime I have set up a small page on our website:
>>>>>>>>>> 
>>>>>>>>>> https://www.ipfire.org/dnsbl
>>>>>>>>>> 
>>>>>>>>>> I would like to run this as a first-class project inside IPFire like we are doing with IPFire Location. That means that we need to tell people about what we are doing. Hopefully this page is a little start.
>>>>>>>>>> 
>>>>>>>>>> Initially it has a couple of high-level bullet points about what we are trying to achieve. I don’t think the text is very good, yet, but it is the best I had in that moment. There is then also a list of the lists that we currently offer. For each list, a detailed page will tell you about the license, how many domains are listed, when the last update has been, the sources and even there is a history page that shows all the changes whenever they have happened.
>>>>>>>>>> 
>>>>>>>>>> Finally there is a section that explains “How To Use?” the list which I would love to extend to include AdGuard Plus and things like that as well as Pi-Hole and whatever else could use the list. In a later step we should go ahead and talk to any projects to include our list(s) into their dropdown so that people can enable them nice and easy.
>>>>>>>>>> 
>>>>>>>>>> Behind the web page there is an API service that is running on the host that is running the DNSBL. The frontend web app that is running www.ipfire.org <http://www.ipfire.org/> is connecting to that API service to fetch the current lists, any details and so on. That way, we can split the logic and avoid creating a huge monolith of a web app. This also means that page could be down a little as I am still working on the entire thing and will frequently restart it.
>>>>>>>>>> 
>>>>>>>>>> The API documentation is available here and the API is publicly available: https://api.dnsbl.ipfire.org/docs
>>>>>>>>>> 
>>>>>>>>>> The website/API allows to file reports for anything that does not seem to be right on any of the lists. I would like to keep it as an open process, however, long-term, this cannot cost us any time. In the current stage, the reports are getting filed and that is about it. I still need to build out some way for admins or moderators (I am not sure what kind of roles I want to have here) to accept or reject those reports.
>>>>>>>>>> 
>>>>>>>>>> In case of us receiving a domain from a source list, I would rather like to submit a report to upstream for them to de-list. That way, we don’t have any admin to do and we are contributing back to other list. That would be a very good thing to do. We cannot however throw tons of emails at some random upstream projects without co-ordinating this first. By not reporting upstream, we will probably over time create large whitelists and I am not sure if that is a good thing to do.
>>>>>>>>>> 
>>>>>>>>>> Finally, there is a search box that can be used to find out if a domain is listed on any of the lists.
>>>>>>>>>> 
>>>>>>>>>>>> If you download and open any of the files, you will see a large header that includes copyright information and lists all sources that have been used to create the individual lists. This way we ensure maximum transparency, comply with the terms of the individual licenses of the source lists and give credit to the people who help us to put together the most perfect list for our users.
>>>>>>>>>>>> 
>>>>>>>>>>>> I would like this to become a project that is not only being used in IPFire. We can and will be compatible with other solutions like AdGuard, PiHole so that people can use our lists if they would like to even though they are not using IPFire. Hopefully, these users will also feed back to us so that we can improve our lists over time and make them one of the best options out there.
>>>>>>>>>>>> 
>>>>>>>>>>>> All lists are available as a simple text file that lists the domains. Then there is a hosts file available as well as a DNS zone file and an RPZ file. Each list is individually available to be used in squidGuard and there is a larger tarball available with all lists that can be used in IPFire’s URL Filter. I am planning to add Suricata/Snort signatures whenever I have time to do so. Even though it is not a good idea to filter pornographic content this way, I suppose that catching malware and blocking DoH are good use-cases for an IPS. Time will tell…
>>>>>>>>>>>> 
>>>>>>>>>>>> As a start, we will make these lists available in IPFire’s URL Filter and collect some feedback about how we are doing. Afterwards, we can see where else we can take this project.
>>>>>>>>>>>> 
>>>>>>>>>>>> If you want to enable this on your system, simply add the URL to your autoupdate.urls file like here:
>>>>>>>>>>>> 
>>>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f
>>>>>>>>>>> I also tested out adding the IPFire url to autoupdate.urls and that also worked fine for me.
>>>>>>>>>> 
>>>>>>>>>> Very good. Should we include this already with Core Update 200? I don’t think we would break anything, but we might already gain a couple more people who are helping us to test this all?
>>>>>>>>> 
>>>>>>>>> I think that would be a good idea.
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> The next step would be to build and test our DNS infrastructure. In the “How To Use?” Section on the pages of the individual lists, you can already see some instructions on how to use the lists as an RPZ. In comparison to other “providers”, I would prefer if people would be using DNS to fetch the lists. This is simply to push out updates in a cheap way for us and also do it very regularly.
>>>>>>>>>> 
>>>>>>>>>> Initially, clients will pull the entire list using AXFR. There is no way around this as they need to have the data in the first place. After that, clients will only need the changes. As you can see in the history, the lists don’t actually change that often. Sometimes only once a day and therefore downloading the entire list again would be a huge waste of data, both on the client side, but also for us hosting then.
>>>>>>>>>> 
>>>>>>>>>> Some other providers update their lists “every 10 minutes”, and there won't be any changes whatsoever. We don’t do that. We will only export the lists again when they have actually changed. The timestamps on the files that we offer using HTTPS can be checked by clients so that they won’t re-download the list again if it has not been changed. But using HTTPS still means that we would have to re-download the entire list and not only the changes.
>>>>>>>>>> 
>>>>>>>>>> Using DNS and IXFR will update the lists by only transferring a few kilobytes and therefore we can have clients check once an hour if a list has actually changed and only send out the raw changes. That way, we will be able to serve millions of clients at very cheap cost and they will always have a very up to date list.
>>>>>>>>>> 
>>>>>>>>>> As far as I can see any DNS software that supports RPZs supports AXFR/IXFR with exception of Knot Resolver which expects the zone to be downloaded externally. There is a ticket for AXFR/IXFR support (https://gitlab.nic.cz/knot/knot-resolver/-/issues/195).
>>>>>>>>>> 
>>>>>>>>>> Initially, some of the lists have been *huge* which is why a simple HTTP download is not feasible. The porn list was over 100 MiB. We could have spent thousands on just traffic alone which I don’t have for this kind of project. It would also be unnecessary money being spent. There are simply better solutions out there. But then I built something that basically tests the data that we are receiving from upstream but simply checking if a listed domain still exists. The result was very astonishing to me.
>>>>>>>>>> 
>>>>>>>>>> So whenever someone adds a domain to the list, we will (eventually, but not immediately) check if we can resolve the domain’s SOA record. If not, we mark the domain as non-active and will no longer include them in the exported data. This brought down the porn list from just under 5 million domains to just 421k. On the sources page (https://www.ipfire.org/dnsbl/lists/porn/sources) I am listing the percentage of dead domains from each of them and the UT1 list has 94% dead domains. Wow.
>>>>>>>>>> 
>>>>>>>>>> If we cannot resolve the domain, neither can our users. So we would otherwise fill the lists with tons of domains that simply could never be reached. And if they cannot be reached, why would we block them? We would waste bandwidth and a lot of memory on each single client.
>>>>>>>>>> 
>>>>>>>>>> The other sources have similarly high rations of dead domains. Most of them are in the 50-80% range. Therefore I am happy that we are doing some extra work here to give our users much better data for their filtering.
>>>>>>>>> 
>>>>>>>>> Removing all dead entries sounds like an excellent step.
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> 
>>>>>>>>> Adolf.
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> So, if you like, please go and check out the RPZ blocking with Unbound. Instructions are on the page. I would be happy to hear how this is turning out.
>>>>>>>>>> 
>>>>>>>>>> Please let me know if there are any more questions, and I would be glad to answer them.
>>>>>>>>>> 
>>>>>>>>>> Happy New Year,
>>>>>>>>>> -Michael
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Regards,
>>>>>>>>>>> Adolf.
>>>>>>>>>>>> This email is just a brain dump from me to this list. I would be happy to answer any questions about implementation details, etc. if people are interested. Right now, this email is long enough already…
>>>>>>>>>>>> 
>>>>>>>>>>>> All the best,
>>>>>>>>>>>> -Michael
>>>>>>>>>>> 
>>>>>>>>>>> -- 
>>>>>>>>>>> Sent from my laptop
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> -- 
>>>>>>> Sent from my laptop
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
> 
> 

Re: Let's launch our own blocklists...

[Michael Tremer]

Hello Adam,

> On 23 Jan 2026, at 19:31, Adam Gibbons <adam.gibbons@ipfire.org> wrote:
> 
> On 22/01/2026 11:33 am, Michael Tremer wrote:
>> Hello everyone,
> 
> Hi Michael and Everyone,
> 
>> Over the past few weeks I have made significant progress on this all, and I think we're getting close to something the community will be really happy with. I'd love to get feedback from the team before we finalise things.
> 
> I've been keeping an eye on the development of IPFire DBL (RIP DNSBL) since the beginning and I agree the community should be very happy with it.
> 
> Having our own trusted, properly maintained blocklists that fit nicely into IPFire (and the wider adblock/domain-blocking ecosystem) makes IPFire a more complete UTM solution while expanding the IPFire brand.
> 
> I think we can make IPFire DBL a trusted alternative to the random lists hosted on the internet these days. Even the small things like DNSSEC by default, removing dead domains,and being able to track additions/removals over time will help build trust.
> 
> With the help of the community we can further refine the lists and make IPFire DBL something that attracts attention to the project as a whole.

Well said.

> 
>> I have added a couple more lists that I thought interesting and I have added a couple more sources that I considered a good start. Hopefully, we will soon gather some more feedback on how well this is all holding up. My main focus has however been on the technology that will power this project.
> 
> I've been using several of the lists for a few weeks now and the number of false-positives has been minimal for my use-case. It has even passed the dreaded "wife test", so I think this is ready for wider testing.

The “wife test” is particularly important for this :)

>> One of the bigger challenges was to create Suricata rules from the lists. Initially I tried to create a ton of rules but since our lists are so large, this quickly became too complicated. I have now settled on using a feature that is only available in more recent versions of Suricata (I believe 7 and later), but since we are already on Suricata 8 in IPFire this won't be a problem for us. All domains for each list are basically compiled into one massively large dataset and one single rule is referring to that dataset. This way, we won't have the option to remove any false-positives, but at least Suricata and the GUI won't starve a really bad death when loading millions of rules.
> 
> Yes, the IPFire DBL rules in IPS currently block all domains on the list without the ability to de-select false-positives. Luckily the reporting feature is very easy to use.
> 
> As mentioned previously, domain blocking in Suricata isn't intended to be the optimal way to handle DNS blocking because connections are dropped rather than returning a DNS response such as NXDOMAIN, so it's best used as a policy backstop.
> 
> For scenarios like schools using the "violence" list this will be perfect, though it might make some websites look/act unusual if the "advertising" list is used. But every scenario is different.
> 
> We see similar effects when people enable the "emerging-dns.rule" ruleset where the ".cc" TLD is blocked by default. Quite often because of the way it's dropped, it's sometimes difficult to diagnose because of the unusual UX. It pops up on the forum occasionally.

You can still have Suricata block the other protocols and disable the DNS rule in each of the categories. That way, you won’t have this problem.

>> Looking ahead, I would like us to start thinking about the RPZ feature that has been on the wishlist. IPFire DBL has been a bigger piece of work, and I think it's worth having a conversation about sustainability. Resources for this need to be allocated and paid for. Open source is about freedom, not free beer — and to keep building features like this, we will need to explore some funding options. I would be interested to hear any ideas you might have that could work for IPFire.
> 
> The community has been seeking a solution like this for some time now. It's great to see the start of what it will end up being. With community support, both funding and false-positive reporting, this can go a long way.

True. I am awaiting their test feedback on this so that we can move forward soon.

> Thank you for the work towards this, Michael.

Pleasure.

> Regards,
> Adam
> 

Re: Let's launch our own blocklists...

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 35982 bytes --]

On 25.01.2026 15:40, Michael Tremer wrote:
> Hello Matthias,

Hi Michael,

> Nice catch!
> 
> I fixed it here and added the missing “;”:

Yep. The missing ";" seems to be fixed, but 'suricata' still doesn't
like our rules. I added 'violence' as an attachment... ;-)

>   https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=775561e322ceed43e255e5547bd76047b9f8a40b
> 
> If you go to the provider settings there is a button to force a ruleset update which should give you the fixed version. Please let me know if this works.

I did that - rules were updated, but no luck:

***SNIP***
...
18:37:37 	suricata: 	[2577] <Info> -- Including configuration file
/var/ipfire/suricata/suricata-used-rulesfiles.yaml.
18:37:37 	suricata: 	[2577] <Error> -- failed to set up dataset 'violence'.
18:37:37 	suricata: 	[2577] <Error> -- error parsing signature "drop dns
any any -> any any (msg:"IPFire DBL [Violence] Blocked DNS Query";
dns.query; domain; dataset:isset,violence,type string,load
datasets/violence.txt; classtype:policy-violation; priority:2;
sid:1048577; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
metadata:dbl violence.dbl.ipfire.org;)" from file
/var/lib/suricata/ipfire_dnsbl-violence.rules at line 39
18:37:37 	suricata: 	[2577] <Error> -- failed to set up dataset 'violence'.
18:37:37 	suricata: 	[2577] <Error> -- error parsing signature "drop
http any any -> any any (msg:"IPFire DBL [Violence] Blocked HTTP
Request"; http.host; domain; dataset:isset,violence,type string,load
datasets/violence.txt; classtype:policy-violation; priority:2;
sid:1048578; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
metadata:dbl violence.dbl.ipfire.org;)" from file
/var/lib/suricata/ipfire_dnsbl-violence.rules at line 40
18:37:37 	suricata: 	[2577] <Error> -- failed to set up dataset 'violence'.
18:37:37 	suricata: 	[2577] <Error> -- error parsing signature "drop tls
any any -> any any (msg:"IPFire DBL [Violence] Blocked TLS Connection";
tls.sni; domain; dataset:isset,violence,type string,load
datasets/violence.txt; classtype:policy-violation; priority:2;
sid:1048579; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
metadata:dbl violence.dbl.ipfire.org;)" from file
/var/lib/suricata/ipfire_dnsbl-violence.rules at line 41
18:37:37 	suricata: 	[2577] <Error> -- failed to set up dataset 'violence'.
18:37:37 	suricata: 	[2577] <Error> -- error parsing signature "drop
quic any any -> any any (msg:"IPFire DBL [Violence] Blocked QUIC
Connection"; quic.sni; domain; dataset:isset,violence,type string,load
datasets/violence.txt; classtype:policy-violation; priority:2;
sid:1048580; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
metadata:dbl violence.dbl.ipfire.org;)" from file
/var/lib/suricata/ipfire_dnsbl-violence.rules at line 42
...
***SNAP***

For better reading - see attached screenshot.

Best
Matthias

> Best,
> -Michael
> 
>> On 24 Jan 2026, at 23:41, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>> 
>> On 23.01.2026 17:39, Michael Tremer wrote:
>>> Hello Matthias,
>> 
>> Hi Michael,
>> 
>>> Thank you very much for testing IPFire DBL.
>> 
>> No problem - I have news:
>> 
>> After taking a closer look to the IPS system logs, unfortunately I found
>> some parsing errors:
>> 
>> 'suricata' complains about missing ";".
>> 
>> ***SNIP***
>> ...
>> 00:32:40 suricata: [13343] <Info> -- Including configuration file
>> /var/ipfire/suricata/suricata-used-rulesfiles.yaml.
>> 00:32:40 suricata: [13343] <Error> -- no terminating ";" found
>> 00:32:40 suricata: [13343] <Error> -- error parsing signature "drop
>> dns any any -> any any (msg:"IPFire DBL [Advertising] Blocked DNS
>> Query"; dns.query; domain; dataset:isset,ads,type string,load
>> datasets/ads.txt; classtype:policy-violation; priority:3; sid:983041;
>> rev:1; reference:url,https://www.ipfire.org/dbl/ads; metadata:dbl
>> ads.dbl.ipfire.org)" from file /var/lib/suricata/ipfire_dnsbl-ads.rules
>> at line 72
>> 00:32:40 suricata: [13343] <Error> -- no terminating ";" found
>> ...
>> ***SNAP***
>> 
>> I tried, but didn't find the right place for any missing ";".
>> 
>> Can "anyone" confirm?
>> 
>> Best
>> Matthias
>> 
>>>> On 23 Jan 2026, at 15:02, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>>>> 
>>>> On 22.01.2026 12:33, Michael Tremer wrote:
>>>>> Hello everyone,
>>>> 
>>>> Hi,
>>>> 
>>>> short feedback from me:
>>>> 
>>>> - I activated both the suricata (IPFire DBL - Domain Blocklist) - and
>>>> the URLfilter lists from 'dbl.ipfire.org'.
>>> 
>>> This is an interesting case. What I didn’t manage to test yet is what happens when Suricata blocks the connection first. If URL Filter sees a domain that is being blocked it will either send you an error page if you are using HTTP, or simply close the connection if it is HTTPS. However, when Suricata comes first in the chain (and it will), it might close the connection because URL Filter has received the request. In the case of HTTPS this does not make any difference because the connection will be closed, but in the HTTP case you won’t see an error page any more and instead have the connection closed, too. You are basically losing the explicit error notification which is a little bit annoying.
>>> 
>>> We could have the same when we are doing the same with Unbound and DNS filtering. Potentially we would need to whitelist the local DNS resolver then, but how is Suricata supposed to know that the same categories are activated in both places?
>>> 
>>>> - I even took the 'smart-tv' domains from the IFire DBL blacklist and
>>>> copied/pasted them in my fritzbox filter lists.
>>> 
>>> LOL Why not use IPFire to filter this as well?
>>> 
>>>> Everything works as expected. Besides, the download of the IPFire
>>>> DBL-list loads a lot faster than the list from 'Univ. Toulouse'... ;-)
>>> 
>>> Yes, we don’t have much traffic on the server, yet.
>>> 
>>>> Functionality is good - no false positives or seen problems. Good work -
>>>> thanks!
>>> 
>>> Nice. We need to distinguish a little between what is a technical issue and what is a false-positive/missing domain on the list. However, testing both at the same time is something we will all cope quite well with :)
>>> 
>>> -Michael
>>> 
>>>> Best
>>>> Matthias
>>>> 
>>>>> Over the past few weeks I have made significant progress on this all, and I think we're getting close to something the community will be really happy with. I'd love to get feedback from the team before we finalise things.
>>>>> 
>>>>> So what has happened?
>>>>> 
>>>>> First of all, the entire project has been renamed. DNSBL is not entirely what this is. Although the lists can be thrown into DNS, they have much more use outside of it that I thought we should simply go with DBL, short for Domain Blocklist. After all, we are only importing domains. The new home of the project therefore is https://www.ipfire.org/dbl
>>>>> 
>>>>> I have added a couple more lists that I thought interesting and I have added a couple more sources that I considered a good start. Hopefully, we will soon gather some more feedback on how well this is all holding up. My main focus has however been on the technology that will power this project.
>>>>> 
>>>>> One of the bigger challenges was to create Suricata rules from the lists. Initially I tried to create a ton of rules but since our lists are so large, this quickly became too complicated. I have now settled on using a feature that is only available in more recent versions of Suricata (I believe 7 and later), but since we are already on Suricata 8 in IPFire this won’t be a problem for us. All domains for each list are basically compiled into one massively large dataset and one single rule is referring to that dataset. This way, we won’t have the option to remove any false-positives, but at least Suricata and the GUI won’t starve a really bad death when loading millions of rules.
>>>>> 
>>>>> Suricata will now be able to use our rules to block access to any listed domains of each of the categories over DNS, HTTP, TLS or QUIC. Although I don’t expect many users to use Suricata to block porn or other things, this is a great backstop to enforce any policy like that. For example, if there is a user on the network who is trying to circumvent the DNS server that might filter out certain domains, even after getting an IP address resolved through other means, they won’t be able to open a TLS/QUIC connection or send a HTTP request to all blocked domains. Some people have said they were interested in blocking DNS-over-HTTPS and this is a perfect way to do this and actually be sure that any server that is being blocked on the list will actually be completely inaccessible.
>>>>> 
>>>>> Those Suricata rules are already available for testing in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=9eb8751487d23dd354a105c28bdbbb0398fe6e85
>>>>> 
>>>>> I have chosen various severities for the lists. If someone was to block advertising using DBL, this is fine, but not a very severe alert. If someone chooses to block malware and there is a system on the network trying to access those domains, this is an alert worth being investigated by an admin. Our new Suricata Reporter will show those violations in different colours based on the severity which helps to identify the right alerts to further investigate.
>>>>> 
>>>>> Formerly I have asked you to test the lists using URL Filter. Those rules are now available as well in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=db160694279a4b10378447f775dd536fdfcfb02a
>>>>> 
>>>>> I talked about a method to remove any dead domains from any sources which is a great way to keep our lists smaller. The pure size of them is a problem in so many ways. That check was however a little bit too ambitious and I had to make it a little bit less eager. Basically if we are in doubt, we need to still list the domain because it might be resolvable by a user.
>>>>> 
>>>>> https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=bb5b6e33b731501d45dea293505f7d42a61d5ce7
>>>>> 
>>>>> So how else could we make the lists smaller without losing any actual data? Since we sometimes list a whole TLD (e.g. .xxx or .porn), there is very little point in listing any domains of this TLD. They will always be caught anyways. So I built a check that marks all domains that don’t need to be included on the exported lists because they will never be needed and was able to shrink the size of the lists by a lot again.
>>>>> 
>>>>> The website does not show this data, but the API returns the number of “subsumed” domains (I didn’t have a better name):
>>>>> 
>>>>> curl https://api.dbl.ipfire.org/lists | jq .
>>>>> 
>>>>> The number shown would normally be added to the total number of domains and usually cuts the size of the list by 50-200%.
>>>>> 
>>>>> Those stats will now also be stored in a history table so that we will be able to track growth of all lists.
>>>>> 
>>>>> Furthermore, the application will now send email notifications for any incoming reports. This way, we will be able to stay in close touch with the reporters and keep them up to date on their submissions as well as inform moderators that there is something to have a look at.
>>>>> 
>>>>> The search has been refactored as well, so that we can show clearly whether something is blocked or not at one glance: https://www.ipfire.org/dbl/search?q=github.com. There is detailed information available on all domains and what happened to them. In case of GitHub.com, this seems to be blocked and unblocked by someone all of the time and we can see a clear audit trail of that: https://www.ipfire.org/dbl/lists/malware/domains/github.com
>>>>> 
>>>>> On the DNS front, I have added some metadata to the zones so that people can programmatically request some data, like when it has been last updated (in a human-friendly timestamp and not only the serial), license, description and so on:
>>>>> 
>>>>> # dig +short ANY _info.ads.dbl.ipfire.org @primary.dbl.ipfire.org
>>>>> "total-domains=42226"
>>>>> "license=CC BY-SA 4.0"
>>>>> "updated-at=2026-01-20T22:17:02.409933+00:00"
>>>>> "description=Blocks domains used for ads, tracking, and ad delivery”
>>>>> 
>>>>> Now, I would like to hear more feedback from you. I know we've all been stretched thin lately, so I especially appreciate anyone who has time to review and provide input. Ideas, just say if you like it or not. Where this could go in the future?
>>>>> 
>>>>> Looking ahead, I would like us to start thinking about the RPZ feature that has been on the wishlist. IPFire DBL has been a bigger piece of work, and I think it's worth having a conversation about sustainability. Resources for this need to be allocated and paid for. Open source is about freedom, not free beer — and to keep building features like this, we will need to explore some funding options. I would be interested to hear any ideas you might have that could work for IPFire.
>>>>> 
>>>>> Please share your thoughts on the mailing list when you can — even a quick 'looks good' or 'I have concerns about X' is valuable. Public discussion helps everyone stay in the loop and contribute.
>>>>> 
>>>>> I am aiming to move forward with this in a week's time, so if you have input, now would be a good time to share it.
>>>>> 
>>>>> Best,
>>>>> -Michael
>>>>> 
>>>>>> On 6 Jan 2026, at 10:20, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>>>>> 
>>>>>> Good Morning Adolf,
>>>>>> 
>>>>>> I had a look at this problem yesterday and it seems that parsing the format is becoming a little bit difficult this way. Since this is only affecting very few domains, I have simply whitelisted them all manually and duckduckgo.com <http://duckduckgo.com/> and others should now be easily reachable again.
>>>>>> 
>>>>>> Please let me know if you have any more findings.
>>>>>> 
>>>>>> All the best,
>>>>>> -Michael
>>>>>> 
>>>>>>> On 5 Jan 2026, at 11:48, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>>>>>> 
>>>>>>> Hello Adolf,
>>>>>>> 
>>>>>>> This is a good find.
>>>>>>> 
>>>>>>> But if duckduckgo.com <http://duckduckgo.com/> is blocked, we will have to have a source somewhere that blocks that domain. Not only a sub-domain of it. Otherwise we have a bug somewhere.
>>>>>>> 
>>>>>>> This is most likely as the domain is listed here, but with some stuff afterwards:
>>>>>>> 
>>>>>>> https://raw.githubusercontent.com/mtxadmin/ublock/refs/heads/master/hosts/_malware_typo
>>>>>>> 
>>>>>>> We strip everything after a # away because we consider it a comment. However, that causes that there is only a line with the domain left which will cause it being listed.
>>>>>>> 
>>>>>>> The # sign is used as some special character but at the same time it is being used for comments.
>>>>>>> 
>>>>>>> I will fix this and then refresh the list.
>>>>>>> 
>>>>>>> -Michael
>>>>>>> 
>>>>>>>> On 5 Jan 2026, at 11:31, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>> 
>>>>>>>> Hi Michael,
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 05/01/2026 12:11, Adolf Belka wrote:
>>>>>>>>> Hi Michael,
>>>>>>>>> 
>>>>>>>>> I have found that the malware list includes duckduckgo.com
>>>>>>>>> 
>>>>>>>> I have checked through the various sources used for the malware list.
>>>>>>>> 
>>>>>>>> The ShadowWhisperer (Tracking) list has improving.duckduckgo.com in its list. I suspect that this one is the one causing the problem.
>>>>>>>> 
>>>>>>>> The mtxadmin (_malware_typo) list has duckduckgo.com mentioned 3 times but not directly as a domain name - looks more like a reference.
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> 
>>>>>>>> Adolf.
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> Adolf.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On 02/01/2026 14:02, Adolf Belka wrote:
>>>>>>>>>> Hi,
>>>>>>>>>> 
>>>>>>>>>> On 02/01/2026 12:09, Michael Tremer wrote:
>>>>>>>>>>> Hello,
>>>>>>>>>>> 
>>>>>>>>>>>> On 30 Dec 2025, at 14:05, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>> 
>>>>>>>>>>>> On 29/12/2025 13:05, Michael Tremer wrote:
>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I hope everyone had a great Christmas and a couple of quiet days to relax from all the stress that was the year 2025.
>>>>>>>>>>>> Still relaxing.
>>>>>>>>>>> 
>>>>>>>>>>> Very good, so let’s have a strong start into 2026 now!
>>>>>>>>>> 
>>>>>>>>>> Starting next week, yes.
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>>> Having a couple of quieter days, I have been working on a new, little (hopefully) side project that has probably been high up on our radar since the Shalla list has shut down in 2020, or maybe even earlier. The goal of the project is to provide good lists with categories of domain names which are usually used to block access to these domains.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> How did we get here?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> As stated before, the URL filter feature in IPFire has the problem that there are not many good blocklists available any more. There used to be a couple more - most famously the Shalla list - but we are now down to a single list from the University of Toulouse. It is a great list, but it is not always the best fit for all users.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Then there has been talk about whether we could implement more blocking features into IPFire that don’t involve the proxy. Most famously blocking over DNS. The problem here remains a the blocking feature is only as good as the data that is fed into it. Some people have been putting forward a number of lists that were suitable for them, but they would not have replaced the blocking functionality as we know it. Their aim is to provide “one list for everything” but that is not what people usually want. It is targeted at a classic home user and the only separation that is being made is any adult/porn/NSFW content which usually is put into a separate list.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> It would have been technically possible to include these lists and let the users decide, but that is not the aim of IPFire. We want to do the job for the user so that their job is getting easier. Including obscure lists that don’t have a clear outline of what they actually want to block (“bad content” is not a category) and passing the burden of figuring out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with cream on top is really not going to work. It is all confusing and will lead to a bad user experience.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> An even bigger problem that is however completely impossible to solve is bad licensing of these lists. A user has asked the publisher of the HaGeZi list whether they could be included in IPFire and under what terms. The response was that the list is available under the terms of the GNU General Public License v3, but that does not seem to be true. The list contains data from various sources. Many of them are licensed under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public agreement that this data may be redistributed, there is a huge legal issue here. We would expose our users to potential copyright infringement which we cannot do under any circumstances. Furthermore many lists are available under a non-commercial license which excludes them from being used in any kind of business. Plenty of IPFire systems are running in businesses, if not even the vast majority.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> In short, these lists are completely unusable for us. Apart from HaGeZi, I consider OISD to have the same problem.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Enough about all the things that are bad. Let’s talk about the new, good things:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Many blacklists on the internet are an amalgamation of other lists. These lists vary in quality with some of them being not that good and without a clear focus and others being excellent data. Since we don’t have the man power to start from scratch, I felt that we can copy the concept that HaGeZi and OISD have started and simply create a new list that is based on other lists at the beginning to have a good starting point. That way, we have much better control over what is going on these lists and we can shape and mould them as we need them. Most importantly, we don’t create a single lists, but many lists that have a clear focus and allow users to choose what they want to block and what not.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> So the current experimental stage that I am in has these lists:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> * Ads
>>>>>>>>>>>>> * Dating
>>>>>>>>>>>>> * DoH
>>>>>>>>>>>>> * Gambling
>>>>>>>>>>>>> * Malware
>>>>>>>>>>>>> * Porn
>>>>>>>>>>>>> * Social
>>>>>>>>>>>>> * Violence
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The categories have been determined by what source lists we have available with good data and are compatible with our chosen license CC BY-SA 4.0. This is the same license that we are using for the IPFire Location database, too.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The main use-cases for any kind of blocking are to comply with legal requirements in networks with children (i.e. schools) to remove any kind of pornographic content, sometimes block social media as well. Gambling and violence are commonly blocked, too. Even more common would be filtering advertising and any malicious content.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The latter is especially difficult because so many source lists throw phishing, spyware, malvertising, tracking and other things into the same bucket. Here this is currently all in the malware list which has therefore become quite large. I am not sure whether this will stay like this in the future or if we will have to make some adjustments, but that is exactly why this is now entering some larger testing.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> What has been built so far? In order to put these lists together properly, track any data about where it is coming from, I have built a tool in Python available here:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> https://git.ipfire.org/?p=dnsbl.git;a=summary
>>>>>>>>>>>>> 
>>>>>>>>>>>>> This tool will automatically update all lists once an hour if there have been any changes and export them in various formats. The exported lists are available for download here:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> https://dnsbl.ipfire.org/lists/
>>>>>>>>>>>> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the custom url works fine.
>>>>>>>>>>>> 
>>>>>>>>>>>> However you need to remember not to put the https:// at the front of the url otherwise the WUI page completes without any error messages but leaves an error message in the system logs saying
>>>>>>>>>>>> 
>>>>>>>>>>>> URL filter blacklist - ERROR: Not a valid URL filter blacklist
>>>>>>>>>>>> 
>>>>>>>>>>>> I found this out the hard way.
>>>>>>>>>>> 
>>>>>>>>>>> Oh yes, I forgot that there is a field on the web UI. If that does not accept https:// as a prefix, please file a bug and we will fix it.
>>>>>>>>>> 
>>>>>>>>>> I will confirm it and raise a bug.
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> The other thing I noticed is that if you already have the Toulouse University list downloaded and you then change to the ipfire custom url then all the existing Toulouse blocklists stay in the directory on IPFire and so you end up with a huge number of category tick boxes, most of which are the old Toulouse ones, which are still available to select and it is not clear which ones are from Toulouse and which ones from IPFire.
>>>>>>>>>>> 
>>>>>>>>>>> Yes, I got the same thing, too. I think this is a bug, too, because otherwise you would have a lot of unused categories lying around that will never be updated. You cannot even tell which ones are from the current list and which ones from the old list.
>>>>>>>>>>> 
>>>>>>>>>>> Long-term we could even consider to remove the Univ. Toulouse list entirely and only have our own lists available which would make the problem go away.
>>>>>>>>>>> 
>>>>>>>>>>>> I think if the blocklist URL source is changed or a custom url is provided the first step should be to remove the old ones already existing.
>>>>>>>>>>>> That might be a problem because users can also create their own blocklists and I believe those go into the same directory.
>>>>>>>>>>> 
>>>>>>>>>>> Good thought. We of course cannot delete the custom lists.
>>>>>>>>>>> 
>>>>>>>>>>>> Without clearing out the old blocklists you end up with a huge number of checkboxes for lists but it is not clear what happens if there is a category that has the same name for the Toulouse list and the IPFire list such as gambling. I will have a look at that and see what happens.
>>>>>>>>>>>> 
>>>>>>>>>>>> Not sure what the best approach to this is.
>>>>>>>>>>> 
>>>>>>>>>>> I believe it is removing all old content.
>>>>>>>>>>> 
>>>>>>>>>>>> Manually deleting all contents of the urlfilter/blacklists/ directory and then selecting the IPFire blocklist url for the custom url I end up with only the 8 categories from the IPFire list.
>>>>>>>>>>>> 
>>>>>>>>>>>> I have tested some gambling sites from the IPFire list and the block worked on some. On others the site no longer exists so there is nothing to block or has been changed to an https site and in that case it went straight through. Also if I chose the http version of the link, it was automatically changed to https and went through without being blocked.
>>>>>>>>>>> 
>>>>>>>>>>> The entire IPFire infrastructure always requires HTTPS. If you start using HTTP, you will be automatically redirected. It is 2026 and we don’t need to talk HTTP any more :)
>>>>>>>>>> 
>>>>>>>>>> Some of the domains in the gambling list (maybe quite a lot) seem to only have an http access. If I tried https it came back with the fact that it couldn't find it.
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> I am glad to hear that the list is actually blocking. It would have been bad if it didn’t. Now we have the big task to check out the “quality” - however that can be determined. I think this is what needs some time…
>>>>>>>>>>> 
>>>>>>>>>>> In the meantime I have set up a small page on our website:
>>>>>>>>>>> 
>>>>>>>>>>> https://www.ipfire.org/dnsbl
>>>>>>>>>>> 
>>>>>>>>>>> I would like to run this as a first-class project inside IPFire like we are doing with IPFire Location. That means that we need to tell people about what we are doing. Hopefully this page is a little start.
>>>>>>>>>>> 
>>>>>>>>>>> Initially it has a couple of high-level bullet points about what we are trying to achieve. I don’t think the text is very good, yet, but it is the best I had in that moment. There is then also a list of the lists that we currently offer. For each list, a detailed page will tell you about the license, how many domains are listed, when the last update has been, the sources and even there is a history page that shows all the changes whenever they have happened.
>>>>>>>>>>> 
>>>>>>>>>>> Finally there is a section that explains “How To Use?” the list which I would love to extend to include AdGuard Plus and things like that as well as Pi-Hole and whatever else could use the list. In a later step we should go ahead and talk to any projects to include our list(s) into their dropdown so that people can enable them nice and easy.
>>>>>>>>>>> 
>>>>>>>>>>> Behind the web page there is an API service that is running on the host that is running the DNSBL. The frontend web app that is running www.ipfire.org <http://www.ipfire.org/> is connecting to that API service to fetch the current lists, any details and so on. That way, we can split the logic and avoid creating a huge monolith of a web app. This also means that page could be down a little as I am still working on the entire thing and will frequently restart it.
>>>>>>>>>>> 
>>>>>>>>>>> The API documentation is available here and the API is publicly available: https://api.dnsbl.ipfire.org/docs
>>>>>>>>>>> 
>>>>>>>>>>> The website/API allows to file reports for anything that does not seem to be right on any of the lists. I would like to keep it as an open process, however, long-term, this cannot cost us any time. In the current stage, the reports are getting filed and that is about it. I still need to build out some way for admins or moderators (I am not sure what kind of roles I want to have here) to accept or reject those reports.
>>>>>>>>>>> 
>>>>>>>>>>> In case of us receiving a domain from a source list, I would rather like to submit a report to upstream for them to de-list. That way, we don’t have any admin to do and we are contributing back to other list. That would be a very good thing to do. We cannot however throw tons of emails at some random upstream projects without co-ordinating this first. By not reporting upstream, we will probably over time create large whitelists and I am not sure if that is a good thing to do.
>>>>>>>>>>> 
>>>>>>>>>>> Finally, there is a search box that can be used to find out if a domain is listed on any of the lists.
>>>>>>>>>>> 
>>>>>>>>>>>>> If you download and open any of the files, you will see a large header that includes copyright information and lists all sources that have been used to create the individual lists. This way we ensure maximum transparency, comply with the terms of the individual licenses of the source lists and give credit to the people who help us to put together the most perfect list for our users.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I would like this to become a project that is not only being used in IPFire. We can and will be compatible with other solutions like AdGuard, PiHole so that people can use our lists if they would like to even though they are not using IPFire. Hopefully, these users will also feed back to us so that we can improve our lists over time and make them one of the best options out there.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> All lists are available as a simple text file that lists the domains. Then there is a hosts file available as well as a DNS zone file and an RPZ file. Each list is individually available to be used in squidGuard and there is a larger tarball available with all lists that can be used in IPFire’s URL Filter. I am planning to add Suricata/Snort signatures whenever I have time to do so. Even though it is not a good idea to filter pornographic content this way, I suppose that catching malware and blocking DoH are good use-cases for an IPS. Time will tell…
>>>>>>>>>>>>> 
>>>>>>>>>>>>> As a start, we will make these lists available in IPFire’s URL Filter and collect some feedback about how we are doing. Afterwards, we can see where else we can take this project.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> If you want to enable this on your system, simply add the URL to your autoupdate.urls file like here:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f
>>>>>>>>>>>> I also tested out adding the IPFire url to autoupdate.urls and that also worked fine for me.
>>>>>>>>>>> 
>>>>>>>>>>> Very good. Should we include this already with Core Update 200? I don’t think we would break anything, but we might already gain a couple more people who are helping us to test this all?
>>>>>>>>>> 
>>>>>>>>>> I think that would be a good idea.
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> The next step would be to build and test our DNS infrastructure. In the “How To Use?” Section on the pages of the individual lists, you can already see some instructions on how to use the lists as an RPZ. In comparison to other “providers”, I would prefer if people would be using DNS to fetch the lists. This is simply to push out updates in a cheap way for us and also do it very regularly.
>>>>>>>>>>> 
>>>>>>>>>>> Initially, clients will pull the entire list using AXFR. There is no way around this as they need to have the data in the first place. After that, clients will only need the changes. As you can see in the history, the lists don’t actually change that often. Sometimes only once a day and therefore downloading the entire list again would be a huge waste of data, both on the client side, but also for us hosting then.
>>>>>>>>>>> 
>>>>>>>>>>> Some other providers update their lists “every 10 minutes”, and there won't be any changes whatsoever. We don’t do that. We will only export the lists again when they have actually changed. The timestamps on the files that we offer using HTTPS can be checked by clients so that they won’t re-download the list again if it has not been changed. But using HTTPS still means that we would have to re-download the entire list and not only the changes.
>>>>>>>>>>> 
>>>>>>>>>>> Using DNS and IXFR will update the lists by only transferring a few kilobytes and therefore we can have clients check once an hour if a list has actually changed and only send out the raw changes. That way, we will be able to serve millions of clients at very cheap cost and they will always have a very up to date list.
>>>>>>>>>>> 
>>>>>>>>>>> As far as I can see any DNS software that supports RPZs supports AXFR/IXFR with exception of Knot Resolver which expects the zone to be downloaded externally. There is a ticket for AXFR/IXFR support (https://gitlab.nic.cz/knot/knot-resolver/-/issues/195).
>>>>>>>>>>> 
>>>>>>>>>>> Initially, some of the lists have been *huge* which is why a simple HTTP download is not feasible. The porn list was over 100 MiB. We could have spent thousands on just traffic alone which I don’t have for this kind of project. It would also be unnecessary money being spent. There are simply better solutions out there. But then I built something that basically tests the data that we are receiving from upstream but simply checking if a listed domain still exists. The result was very astonishing to me.
>>>>>>>>>>> 
>>>>>>>>>>> So whenever someone adds a domain to the list, we will (eventually, but not immediately) check if we can resolve the domain’s SOA record. If not, we mark the domain as non-active and will no longer include them in the exported data. This brought down the porn list from just under 5 million domains to just 421k. On the sources page (https://www.ipfire.org/dnsbl/lists/porn/sources) I am listing the percentage of dead domains from each of them and the UT1 list has 94% dead domains. Wow.
>>>>>>>>>>> 
>>>>>>>>>>> If we cannot resolve the domain, neither can our users. So we would otherwise fill the lists with tons of domains that simply could never be reached. And if they cannot be reached, why would we block them? We would waste bandwidth and a lot of memory on each single client.
>>>>>>>>>>> 
>>>>>>>>>>> The other sources have similarly high rations of dead domains. Most of them are in the 50-80% range. Therefore I am happy that we are doing some extra work here to give our users much better data for their filtering.
>>>>>>>>>> 
>>>>>>>>>> Removing all dead entries sounds like an excellent step.
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> 
>>>>>>>>>> Adolf.
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> So, if you like, please go and check out the RPZ blocking with Unbound. Instructions are on the page. I would be happy to hear how this is turning out.
>>>>>>>>>>> 
>>>>>>>>>>> Please let me know if there are any more questions, and I would be glad to answer them.
>>>>>>>>>>> 
>>>>>>>>>>> Happy New Year,
>>>>>>>>>>> -Michael
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>> This email is just a brain dump from me to this list. I would be happy to answer any questions about implementation details, etc. if people are interested. Right now, this email is long enough already…
>>>>>>>>>>>>> 
>>>>>>>>>>>>> All the best,
>>>>>>>>>>>>> -Michael
>>>>>>>>>>>> 
>>>>>>>>>>>> -- 
>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> -- 
>>>>>>>> Sent from my laptop
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>> 
>> 
> 

[-- Attachment #2: suricata-log.jpg --]
[-- Type: image/jpeg, Size: 254049 bytes --]

[-- Attachment #3: ipfire_dnsbl-violence.rules --]
[-- Type: text/plain, Size: 2708 bytes --]

###############################################################################
# Domain Blocklist for IPFire
###############################################################################
#
# This file contains domains that are part of the official IPFire Domain
# Blocklist Project. It is intended for use with DNS servers and firewalls
# that support blocklisting of known malicious or unwanted domains.
#
# List          : Violence
#
#  Blocks domains featuring violent or extremist content
#
# License       : CC BY-SA 4.0
# Updated       : 2026-01-24T22:17:03.022220+00:00
# Total Entries : 264
#
# Copyright (C) 2026 - IPFire Team
#
# For more information or to contribute:
#   https://dbl.ipfire.org/
#
###############################################################################
#
# This list contains data from these sources:
#
#   * ShadowWhisperer - Shock (Public Domain)
#     https://raw.githubusercontent.com/ShadowWhisperer/BlockLists/master/Lists/Shock
#     Updated: 2026-01-03T13:30:59.055699+00:00
#   * Sinfonietta (MIT)
#     https://github.com/Sinfonietta/hostfiles/raw/refs/heads/master/snuff-hosts
#     Updated: 2025-12-31T14:55:55.534437+00:00
#   * UT1 (CC-BY-SA 4.0)
#     https://dsi.ut-capitole.fr/blacklists/download/aggressive.tar.gz
#     Updated: 2026-01-24T22:17:03.022220+00:00
#   * UT1 (CC-BY-SA 4.0)
#     https://dsi.ut-capitole.fr/blacklists/download/violence.tar.gz
#     Updated: 2026-01-24T22:17:03.022220+00:00
#
drop dns any any -> any any (msg:"IPFire DBL [Violence] Blocked DNS Query"; dns.query; domain; dataset:isset,violence,type string,load datasets/violence.txt; classtype:policy-violation; priority:2; sid:1048577; rev:1; reference:url,https://www.ipfire.org/dbl/violence; metadata:dbl violence.dbl.ipfire.org;)
drop http any any -> any any (msg:"IPFire DBL [Violence] Blocked HTTP Request"; http.host; domain; dataset:isset,violence,type string,load datasets/violence.txt; classtype:policy-violation; priority:2; sid:1048578; rev:1; reference:url,https://www.ipfire.org/dbl/violence; metadata:dbl violence.dbl.ipfire.org;)
drop tls any any -> any any (msg:"IPFire DBL [Violence] Blocked TLS Connection"; tls.sni; domain; dataset:isset,violence,type string,load datasets/violence.txt; classtype:policy-violation; priority:2; sid:1048579; rev:1; reference:url,https://www.ipfire.org/dbl/violence; metadata:dbl violence.dbl.ipfire.org;)
drop quic any any -> any any (msg:"IPFire DBL [Violence] Blocked QUIC Connection"; quic.sni; domain; dataset:isset,violence,type string,load datasets/violence.txt; classtype:policy-violation; priority:2; sid:1048580; rev:1; reference:url,https://www.ipfire.org/dbl/violence; metadata:dbl violence.dbl.ipfire.org;)

Re: Let's launch our own blocklists...

[Michael Tremer]

Hello Matthias,

Are you running this on Core Update 200?

There were some changes required so that we will extract the datasets from the rules tarballs. I am using a special feature in Suricata so that the lists won’t be using too much memory and will be quickly searchable:

  https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=f0b43241a501f7c545e3cb15f6989e945c60b3e2

-Michael

> On 25 Jan 2026, at 17:50, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
> 
> On 25.01.2026 15:40, Michael Tremer wrote:
>> Hello Matthias,
> 
> Hi Michael,
> 
>> Nice catch!
>> 
>> I fixed it here and added the missing “;”:
> 
> Yep. The missing ";" seems to be fixed, but 'suricata' still doesn't
> like our rules. I added 'violence' as an attachment... ;-)
> 
>>  https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=775561e322ceed43e255e5547bd76047b9f8a40b
>> 
>> If you go to the provider settings there is a button to force a ruleset update which should give you the fixed version. Please let me know if this works.
> 
> I did that - rules were updated, but no luck:
> 
> ***SNIP***
> ...
> 18:37:37  suricata:  [2577] <Info> -- Including configuration file
> /var/ipfire/suricata/suricata-used-rulesfiles.yaml.
> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop dns
> any any -> any any (msg:"IPFire DBL [Violence] Blocked DNS Query";
> dns.query; domain; dataset:isset,violence,type string,load
> datasets/violence.txt; classtype:policy-violation; priority:2;
> sid:1048577; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
> metadata:dbl violence.dbl.ipfire.org;)" from file
> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 39
> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop
> http any any -> any any (msg:"IPFire DBL [Violence] Blocked HTTP
> Request"; http.host; domain; dataset:isset,violence,type string,load
> datasets/violence.txt; classtype:policy-violation; priority:2;
> sid:1048578; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
> metadata:dbl violence.dbl.ipfire.org;)" from file
> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 40
> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop tls
> any any -> any any (msg:"IPFire DBL [Violence] Blocked TLS Connection";
> tls.sni; domain; dataset:isset,violence,type string,load
> datasets/violence.txt; classtype:policy-violation; priority:2;
> sid:1048579; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
> metadata:dbl violence.dbl.ipfire.org;)" from file
> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 41
> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop
> quic any any -> any any (msg:"IPFire DBL [Violence] Blocked QUIC
> Connection"; quic.sni; domain; dataset:isset,violence,type string,load
> datasets/violence.txt; classtype:policy-violation; priority:2;
> sid:1048580; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
> metadata:dbl violence.dbl.ipfire.org;)" from file
> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 42
> ...
> ***SNAP***
> 
> For better reading - see attached screenshot.
> 
> Best
> Matthias
> 
>> Best,
>> -Michael
>> 
>>> On 24 Jan 2026, at 23:41, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>>> 
>>> On 23.01.2026 17:39, Michael Tremer wrote:
>>>> Hello Matthias,
>>> 
>>> Hi Michael,
>>> 
>>>> Thank you very much for testing IPFire DBL.
>>> 
>>> No problem - I have news:
>>> 
>>> After taking a closer look to the IPS system logs, unfortunately I found
>>> some parsing errors:
>>> 
>>> 'suricata' complains about missing ";".
>>> 
>>> ***SNIP***
>>> ...
>>> 00:32:40 suricata: [13343] <Info> -- Including configuration file
>>> /var/ipfire/suricata/suricata-used-rulesfiles.yaml.
>>> 00:32:40 suricata: [13343] <Error> -- no terminating ";" found
>>> 00:32:40 suricata: [13343] <Error> -- error parsing signature "drop
>>> dns any any -> any any (msg:"IPFire DBL [Advertising] Blocked DNS
>>> Query"; dns.query; domain; dataset:isset,ads,type string,load
>>> datasets/ads.txt; classtype:policy-violation; priority:3; sid:983041;
>>> rev:1; reference:url,https://www.ipfire.org/dbl/ads; metadata:dbl
>>> ads.dbl.ipfire.org)" from file /var/lib/suricata/ipfire_dnsbl-ads.rules
>>> at line 72
>>> 00:32:40 suricata: [13343] <Error> -- no terminating ";" found
>>> ...
>>> ***SNAP***
>>> 
>>> I tried, but didn't find the right place for any missing ";".
>>> 
>>> Can "anyone" confirm?
>>> 
>>> Best
>>> Matthias
>>> 
>>>>> On 23 Jan 2026, at 15:02, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>>>>> 
>>>>> On 22.01.2026 12:33, Michael Tremer wrote:
>>>>>> Hello everyone,
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> short feedback from me:
>>>>> 
>>>>> - I activated both the suricata (IPFire DBL - Domain Blocklist) - and
>>>>> the URLfilter lists from 'dbl.ipfire.org'.
>>>> 
>>>> This is an interesting case. What I didn’t manage to test yet is what happens when Suricata blocks the connection first. If URL Filter sees a domain that is being blocked it will either send you an error page if you are using HTTP, or simply close the connection if it is HTTPS. However, when Suricata comes first in the chain (and it will), it might close the connection because URL Filter has received the request. In the case of HTTPS this does not make any difference because the connection will be closed, but in the HTTP case you won’t see an error page any more and instead have the connection closed, too. You are basically losing the explicit error notification which is a little bit annoying.
>>>> 
>>>> We could have the same when we are doing the same with Unbound and DNS filtering. Potentially we would need to whitelist the local DNS resolver then, but how is Suricata supposed to know that the same categories are activated in both places?
>>>> 
>>>>> - I even took the 'smart-tv' domains from the IFire DBL blacklist and
>>>>> copied/pasted them in my fritzbox filter lists.
>>>> 
>>>> LOL Why not use IPFire to filter this as well?
>>>> 
>>>>> Everything works as expected. Besides, the download of the IPFire
>>>>> DBL-list loads a lot faster than the list from 'Univ. Toulouse'... ;-)
>>>> 
>>>> Yes, we don’t have much traffic on the server, yet.
>>>> 
>>>>> Functionality is good - no false positives or seen problems. Good work -
>>>>> thanks!
>>>> 
>>>> Nice. We need to distinguish a little between what is a technical issue and what is a false-positive/missing domain on the list. However, testing both at the same time is something we will all cope quite well with :)
>>>> 
>>>> -Michael
>>>> 
>>>>> Best
>>>>> Matthias
>>>>> 
>>>>>> Over the past few weeks I have made significant progress on this all, and I think we're getting close to something the community will be really happy with. I'd love to get feedback from the team before we finalise things.
>>>>>> 
>>>>>> So what has happened?
>>>>>> 
>>>>>> First of all, the entire project has been renamed. DNSBL is not entirely what this is. Although the lists can be thrown into DNS, they have much more use outside of it that I thought we should simply go with DBL, short for Domain Blocklist. After all, we are only importing domains. The new home of the project therefore is https://www.ipfire.org/dbl
>>>>>> 
>>>>>> I have added a couple more lists that I thought interesting and I have added a couple more sources that I considered a good start. Hopefully, we will soon gather some more feedback on how well this is all holding up. My main focus has however been on the technology that will power this project.
>>>>>> 
>>>>>> One of the bigger challenges was to create Suricata rules from the lists. Initially I tried to create a ton of rules but since our lists are so large, this quickly became too complicated. I have now settled on using a feature that is only available in more recent versions of Suricata (I believe 7 and later), but since we are already on Suricata 8 in IPFire this won’t be a problem for us. All domains for each list are basically compiled into one massively large dataset and one single rule is referring to that dataset. This way, we won’t have the option to remove any false-positives, but at least Suricata and the GUI won’t starve a really bad death when loading millions of rules.
>>>>>> 
>>>>>> Suricata will now be able to use our rules to block access to any listed domains of each of the categories over DNS, HTTP, TLS or QUIC. Although I don’t expect many users to use Suricata to block porn or other things, this is a great backstop to enforce any policy like that. For example, if there is a user on the network who is trying to circumvent the DNS server that might filter out certain domains, even after getting an IP address resolved through other means, they won’t be able to open a TLS/QUIC connection or send a HTTP request to all blocked domains. Some people have said they were interested in blocking DNS-over-HTTPS and this is a perfect way to do this and actually be sure that any server that is being blocked on the list will actually be completely inaccessible.
>>>>>> 
>>>>>> Those Suricata rules are already available for testing in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=9eb8751487d23dd354a105c28bdbbb0398fe6e85
>>>>>> 
>>>>>> I have chosen various severities for the lists. If someone was to block advertising using DBL, this is fine, but not a very severe alert. If someone chooses to block malware and there is a system on the network trying to access those domains, this is an alert worth being investigated by an admin. Our new Suricata Reporter will show those violations in different colours based on the severity which helps to identify the right alerts to further investigate.
>>>>>> 
>>>>>> Formerly I have asked you to test the lists using URL Filter. Those rules are now available as well in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=db160694279a4b10378447f775dd536fdfcfb02a
>>>>>> 
>>>>>> I talked about a method to remove any dead domains from any sources which is a great way to keep our lists smaller. The pure size of them is a problem in so many ways. That check was however a little bit too ambitious and I had to make it a little bit less eager. Basically if we are in doubt, we need to still list the domain because it might be resolvable by a user.
>>>>>> 
>>>>>> https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=bb5b6e33b731501d45dea293505f7d42a61d5ce7
>>>>>> 
>>>>>> So how else could we make the lists smaller without losing any actual data? Since we sometimes list a whole TLD (e.g. .xxx or .porn), there is very little point in listing any domains of this TLD. They will always be caught anyways. So I built a check that marks all domains that don’t need to be included on the exported lists because they will never be needed and was able to shrink the size of the lists by a lot again.
>>>>>> 
>>>>>> The website does not show this data, but the API returns the number of “subsumed” domains (I didn’t have a better name):
>>>>>> 
>>>>>> curl https://api.dbl.ipfire.org/lists | jq .
>>>>>> 
>>>>>> The number shown would normally be added to the total number of domains and usually cuts the size of the list by 50-200%.
>>>>>> 
>>>>>> Those stats will now also be stored in a history table so that we will be able to track growth of all lists.
>>>>>> 
>>>>>> Furthermore, the application will now send email notifications for any incoming reports. This way, we will be able to stay in close touch with the reporters and keep them up to date on their submissions as well as inform moderators that there is something to have a look at.
>>>>>> 
>>>>>> The search has been refactored as well, so that we can show clearly whether something is blocked or not at one glance: https://www.ipfire.org/dbl/search?q=github.com. There is detailed information available on all domains and what happened to them. In case of GitHub.com, this seems to be blocked and unblocked by someone all of the time and we can see a clear audit trail of that: https://www.ipfire.org/dbl/lists/malware/domains/github.com
>>>>>> 
>>>>>> On the DNS front, I have added some metadata to the zones so that people can programmatically request some data, like when it has been last updated (in a human-friendly timestamp and not only the serial), license, description and so on:
>>>>>> 
>>>>>> # dig +short ANY _info.ads.dbl.ipfire.org @primary.dbl.ipfire.org
>>>>>> "total-domains=42226"
>>>>>> "license=CC BY-SA 4.0"
>>>>>> "updated-at=2026-01-20T22:17:02.409933+00:00"
>>>>>> "description=Blocks domains used for ads, tracking, and ad delivery”
>>>>>> 
>>>>>> Now, I would like to hear more feedback from you. I know we've all been stretched thin lately, so I especially appreciate anyone who has time to review and provide input. Ideas, just say if you like it or not. Where this could go in the future?
>>>>>> 
>>>>>> Looking ahead, I would like us to start thinking about the RPZ feature that has been on the wishlist. IPFire DBL has been a bigger piece of work, and I think it's worth having a conversation about sustainability. Resources for this need to be allocated and paid for. Open source is about freedom, not free beer — and to keep building features like this, we will need to explore some funding options. I would be interested to hear any ideas you might have that could work for IPFire.
>>>>>> 
>>>>>> Please share your thoughts on the mailing list when you can — even a quick 'looks good' or 'I have concerns about X' is valuable. Public discussion helps everyone stay in the loop and contribute.
>>>>>> 
>>>>>> I am aiming to move forward with this in a week's time, so if you have input, now would be a good time to share it.
>>>>>> 
>>>>>> Best,
>>>>>> -Michael
>>>>>> 
>>>>>>> On 6 Jan 2026, at 10:20, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>>>>>> 
>>>>>>> Good Morning Adolf,
>>>>>>> 
>>>>>>> I had a look at this problem yesterday and it seems that parsing the format is becoming a little bit difficult this way. Since this is only affecting very few domains, I have simply whitelisted them all manually and duckduckgo.com <http://duckduckgo.com/> and others should now be easily reachable again.
>>>>>>> 
>>>>>>> Please let me know if you have any more findings.
>>>>>>> 
>>>>>>> All the best,
>>>>>>> -Michael
>>>>>>> 
>>>>>>>> On 5 Jan 2026, at 11:48, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>>>>>>> 
>>>>>>>> Hello Adolf,
>>>>>>>> 
>>>>>>>> This is a good find.
>>>>>>>> 
>>>>>>>> But if duckduckgo.com <http://duckduckgo.com/> is blocked, we will have to have a source somewhere that blocks that domain. Not only a sub-domain of it. Otherwise we have a bug somewhere.
>>>>>>>> 
>>>>>>>> This is most likely as the domain is listed here, but with some stuff afterwards:
>>>>>>>> 
>>>>>>>> https://raw.githubusercontent.com/mtxadmin/ublock/refs/heads/master/hosts/_malware_typo
>>>>>>>> 
>>>>>>>> We strip everything after a # away because we consider it a comment. However, that causes that there is only a line with the domain left which will cause it being listed.
>>>>>>>> 
>>>>>>>> The # sign is used as some special character but at the same time it is being used for comments.
>>>>>>>> 
>>>>>>>> I will fix this and then refresh the list.
>>>>>>>> 
>>>>>>>> -Michael
>>>>>>>> 
>>>>>>>>> On 5 Jan 2026, at 11:31, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>> 
>>>>>>>>> Hi Michael,
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On 05/01/2026 12:11, Adolf Belka wrote:
>>>>>>>>>> Hi Michael,
>>>>>>>>>> 
>>>>>>>>>> I have found that the malware list includes duckduckgo.com
>>>>>>>>>> 
>>>>>>>>> I have checked through the various sources used for the malware list.
>>>>>>>>> 
>>>>>>>>> The ShadowWhisperer (Tracking) list has improving.duckduckgo.com in its list. I suspect that this one is the one causing the problem.
>>>>>>>>> 
>>>>>>>>> The mtxadmin (_malware_typo) list has duckduckgo.com mentioned 3 times but not directly as a domain name - looks more like a reference.
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> 
>>>>>>>>> Adolf.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> Adolf.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On 02/01/2026 14:02, Adolf Belka wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>> 
>>>>>>>>>>> On 02/01/2026 12:09, Michael Tremer wrote:
>>>>>>>>>>>> Hello,
>>>>>>>>>>>> 
>>>>>>>>>>>>> On 30 Dec 2025, at 14:05, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 29/12/2025 13:05, Michael Tremer wrote:
>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I hope everyone had a great Christmas and a couple of quiet days to relax from all the stress that was the year 2025.
>>>>>>>>>>>>> Still relaxing.
>>>>>>>>>>>> 
>>>>>>>>>>>> Very good, so let’s have a strong start into 2026 now!
>>>>>>>>>>> 
>>>>>>>>>>> Starting next week, yes.
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>>> Having a couple of quieter days, I have been working on a new, little (hopefully) side project that has probably been high up on our radar since the Shalla list has shut down in 2020, or maybe even earlier. The goal of the project is to provide good lists with categories of domain names which are usually used to block access to these domains.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> How did we get here?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> As stated before, the URL filter feature in IPFire has the problem that there are not many good blocklists available any more. There used to be a couple more - most famously the Shalla list - but we are now down to a single list from the University of Toulouse. It is a great list, but it is not always the best fit for all users.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Then there has been talk about whether we could implement more blocking features into IPFire that don’t involve the proxy. Most famously blocking over DNS. The problem here remains a the blocking feature is only as good as the data that is fed into it. Some people have been putting forward a number of lists that were suitable for them, but they would not have replaced the blocking functionality as we know it. Their aim is to provide “one list for everything” but that is not what people usually want. It is targeted at a classic home user and the only separation that is being made is any adult/porn/NSFW content which usually is put into a separate list.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> It would have been technically possible to include these lists and let the users decide, but that is not the aim of IPFire. We want to do the job for the user so that their job is getting easier. Including obscure lists that don’t have a clear outline of what they actually want to block (“bad content” is not a category) and passing the burden of figuring out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with cream on top is really not going to work. It is all confusing and will lead to a bad user experience.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> An even bigger problem that is however completely impossible to solve is bad licensing of these lists. A user has asked the publisher of the HaGeZi list whether they could be included in IPFire and under what terms. The response was that the list is available under the terms of the GNU General Public License v3, but that does not seem to be true. The list contains data from various sources. Many of them are licensed under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public agreement that this data may be redistributed, there is a huge legal issue here. We would expose our users to potential copyright infringement which we cannot do under any circumstances. Furthermore many lists are available under a non-commercial license which excludes them from being used in any kind of business. Plenty of IPFire systems are running in businesses, if not even the vast majority.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> In short, these lists are completely unusable for us. Apart from HaGeZi, I consider OISD to have the same problem.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Enough about all the things that are bad. Let’s talk about the new, good things:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Many blacklists on the internet are an amalgamation of other lists. These lists vary in quality with some of them being not that good and without a clear focus and others being excellent data. Since we don’t have the man power to start from scratch, I felt that we can copy the concept that HaGeZi and OISD have started and simply create a new list that is based on other lists at the beginning to have a good starting point. That way, we have much better control over what is going on these lists and we can shape and mould them as we need them. Most importantly, we don’t create a single lists, but many lists that have a clear focus and allow users to choose what they want to block and what not.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> So the current experimental stage that I am in has these lists:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> * Ads
>>>>>>>>>>>>>> * Dating
>>>>>>>>>>>>>> * DoH
>>>>>>>>>>>>>> * Gambling
>>>>>>>>>>>>>> * Malware
>>>>>>>>>>>>>> * Porn
>>>>>>>>>>>>>> * Social
>>>>>>>>>>>>>> * Violence
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The categories have been determined by what source lists we have available with good data and are compatible with our chosen license CC BY-SA 4.0. This is the same license that we are using for the IPFire Location database, too.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The main use-cases for any kind of blocking are to comply with legal requirements in networks with children (i.e. schools) to remove any kind of pornographic content, sometimes block social media as well. Gambling and violence are commonly blocked, too. Even more common would be filtering advertising and any malicious content.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The latter is especially difficult because so many source lists throw phishing, spyware, malvertising, tracking and other things into the same bucket. Here this is currently all in the malware list which has therefore become quite large. I am not sure whether this will stay like this in the future or if we will have to make some adjustments, but that is exactly why this is now entering some larger testing.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> What has been built so far? In order to put these lists together properly, track any data about where it is coming from, I have built a tool in Python available here:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> https://git.ipfire.org/?p=dnsbl.git;a=summary
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> This tool will automatically update all lists once an hour if there have been any changes and export them in various formats. The exported lists are available for download here:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> https://dnsbl.ipfire.org/lists/
>>>>>>>>>>>>> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the custom url works fine.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> However you need to remember not to put the https:// at the front of the url otherwise the WUI page completes without any error messages but leaves an error message in the system logs saying
>>>>>>>>>>>>> 
>>>>>>>>>>>>> URL filter blacklist - ERROR: Not a valid URL filter blacklist
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I found this out the hard way.
>>>>>>>>>>>> 
>>>>>>>>>>>> Oh yes, I forgot that there is a field on the web UI. If that does not accept https:// as a prefix, please file a bug and we will fix it.
>>>>>>>>>>> 
>>>>>>>>>>> I will confirm it and raise a bug.
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> The other thing I noticed is that if you already have the Toulouse University list downloaded and you then change to the ipfire custom url then all the existing Toulouse blocklists stay in the directory on IPFire and so you end up with a huge number of category tick boxes, most of which are the old Toulouse ones, which are still available to select and it is not clear which ones are from Toulouse and which ones from IPFire.
>>>>>>>>>>>> 
>>>>>>>>>>>> Yes, I got the same thing, too. I think this is a bug, too, because otherwise you would have a lot of unused categories lying around that will never be updated. You cannot even tell which ones are from the current list and which ones from the old list.
>>>>>>>>>>>> 
>>>>>>>>>>>> Long-term we could even consider to remove the Univ. Toulouse list entirely and only have our own lists available which would make the problem go away.
>>>>>>>>>>>> 
>>>>>>>>>>>>> I think if the blocklist URL source is changed or a custom url is provided the first step should be to remove the old ones already existing.
>>>>>>>>>>>>> That might be a problem because users can also create their own blocklists and I believe those go into the same directory.
>>>>>>>>>>>> 
>>>>>>>>>>>> Good thought. We of course cannot delete the custom lists.
>>>>>>>>>>>> 
>>>>>>>>>>>>> Without clearing out the old blocklists you end up with a huge number of checkboxes for lists but it is not clear what happens if there is a category that has the same name for the Toulouse list and the IPFire list such as gambling. I will have a look at that and see what happens.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Not sure what the best approach to this is.
>>>>>>>>>>>> 
>>>>>>>>>>>> I believe it is removing all old content.
>>>>>>>>>>>> 
>>>>>>>>>>>>> Manually deleting all contents of the urlfilter/blacklists/ directory and then selecting the IPFire blocklist url for the custom url I end up with only the 8 categories from the IPFire list.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I have tested some gambling sites from the IPFire list and the block worked on some. On others the site no longer exists so there is nothing to block or has been changed to an https site and in that case it went straight through. Also if I chose the http version of the link, it was automatically changed to https and went through without being blocked.
>>>>>>>>>>>> 
>>>>>>>>>>>> The entire IPFire infrastructure always requires HTTPS. If you start using HTTP, you will be automatically redirected. It is 2026 and we don’t need to talk HTTP any more :)
>>>>>>>>>>> 
>>>>>>>>>>> Some of the domains in the gambling list (maybe quite a lot) seem to only have an http access. If I tried https it came back with the fact that it couldn't find it.
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> I am glad to hear that the list is actually blocking. It would have been bad if it didn’t. Now we have the big task to check out the “quality” - however that can be determined. I think this is what needs some time…
>>>>>>>>>>>> 
>>>>>>>>>>>> In the meantime I have set up a small page on our website:
>>>>>>>>>>>> 
>>>>>>>>>>>> https://www.ipfire.org/dnsbl
>>>>>>>>>>>> 
>>>>>>>>>>>> I would like to run this as a first-class project inside IPFire like we are doing with IPFire Location. That means that we need to tell people about what we are doing. Hopefully this page is a little start.
>>>>>>>>>>>> 
>>>>>>>>>>>> Initially it has a couple of high-level bullet points about what we are trying to achieve. I don’t think the text is very good, yet, but it is the best I had in that moment. There is then also a list of the lists that we currently offer. For each list, a detailed page will tell you about the license, how many domains are listed, when the last update has been, the sources and even there is a history page that shows all the changes whenever they have happened.
>>>>>>>>>>>> 
>>>>>>>>>>>> Finally there is a section that explains “How To Use?” the list which I would love to extend to include AdGuard Plus and things like that as well as Pi-Hole and whatever else could use the list. In a later step we should go ahead and talk to any projects to include our list(s) into their dropdown so that people can enable them nice and easy.
>>>>>>>>>>>> 
>>>>>>>>>>>> Behind the web page there is an API service that is running on the host that is running the DNSBL. The frontend web app that is running www.ipfire.org <http://www.ipfire.org/> is connecting to that API service to fetch the current lists, any details and so on. That way, we can split the logic and avoid creating a huge monolith of a web app. This also means that page could be down a little as I am still working on the entire thing and will frequently restart it.
>>>>>>>>>>>> 
>>>>>>>>>>>> The API documentation is available here and the API is publicly available: https://api.dnsbl.ipfire.org/docs
>>>>>>>>>>>> 
>>>>>>>>>>>> The website/API allows to file reports for anything that does not seem to be right on any of the lists. I would like to keep it as an open process, however, long-term, this cannot cost us any time. In the current stage, the reports are getting filed and that is about it. I still need to build out some way for admins or moderators (I am not sure what kind of roles I want to have here) to accept or reject those reports.
>>>>>>>>>>>> 
>>>>>>>>>>>> In case of us receiving a domain from a source list, I would rather like to submit a report to upstream for them to de-list. That way, we don’t have any admin to do and we are contributing back to other list. That would be a very good thing to do. We cannot however throw tons of emails at some random upstream projects without co-ordinating this first. By not reporting upstream, we will probably over time create large whitelists and I am not sure if that is a good thing to do.
>>>>>>>>>>>> 
>>>>>>>>>>>> Finally, there is a search box that can be used to find out if a domain is listed on any of the lists.
>>>>>>>>>>>> 
>>>>>>>>>>>>>> If you download and open any of the files, you will see a large header that includes copyright information and lists all sources that have been used to create the individual lists. This way we ensure maximum transparency, comply with the terms of the individual licenses of the source lists and give credit to the people who help us to put together the most perfect list for our users.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I would like this to become a project that is not only being used in IPFire. We can and will be compatible with other solutions like AdGuard, PiHole so that people can use our lists if they would like to even though they are not using IPFire. Hopefully, these users will also feed back to us so that we can improve our lists over time and make them one of the best options out there.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> All lists are available as a simple text file that lists the domains. Then there is a hosts file available as well as a DNS zone file and an RPZ file. Each list is individually available to be used in squidGuard and there is a larger tarball available with all lists that can be used in IPFire’s URL Filter. I am planning to add Suricata/Snort signatures whenever I have time to do so. Even though it is not a good idea to filter pornographic content this way, I suppose that catching malware and blocking DoH are good use-cases for an IPS. Time will tell…
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> As a start, we will make these lists available in IPFire’s URL Filter and collect some feedback about how we are doing. Afterwards, we can see where else we can take this project.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> If you want to enable this on your system, simply add the URL to your autoupdate.urls file like here:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f
>>>>>>>>>>>>> I also tested out adding the IPFire url to autoupdate.urls and that also worked fine for me.
>>>>>>>>>>>> 
>>>>>>>>>>>> Very good. Should we include this already with Core Update 200? I don’t think we would break anything, but we might already gain a couple more people who are helping us to test this all?
>>>>>>>>>>> 
>>>>>>>>>>> I think that would be a good idea.
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> The next step would be to build and test our DNS infrastructure. In the “How To Use?” Section on the pages of the individual lists, you can already see some instructions on how to use the lists as an RPZ. In comparison to other “providers”, I would prefer if people would be using DNS to fetch the lists. This is simply to push out updates in a cheap way for us and also do it very regularly.
>>>>>>>>>>>> 
>>>>>>>>>>>> Initially, clients will pull the entire list using AXFR. There is no way around this as they need to have the data in the first place. After that, clients will only need the changes. As you can see in the history, the lists don’t actually change that often. Sometimes only once a day and therefore downloading the entire list again would be a huge waste of data, both on the client side, but also for us hosting then.
>>>>>>>>>>>> 
>>>>>>>>>>>> Some other providers update their lists “every 10 minutes”, and there won't be any changes whatsoever. We don’t do that. We will only export the lists again when they have actually changed. The timestamps on the files that we offer using HTTPS can be checked by clients so that they won’t re-download the list again if it has not been changed. But using HTTPS still means that we would have to re-download the entire list and not only the changes.
>>>>>>>>>>>> 
>>>>>>>>>>>> Using DNS and IXFR will update the lists by only transferring a few kilobytes and therefore we can have clients check once an hour if a list has actually changed and only send out the raw changes. That way, we will be able to serve millions of clients at very cheap cost and they will always have a very up to date list.
>>>>>>>>>>>> 
>>>>>>>>>>>> As far as I can see any DNS software that supports RPZs supports AXFR/IXFR with exception of Knot Resolver which expects the zone to be downloaded externally. There is a ticket for AXFR/IXFR support (https://gitlab.nic.cz/knot/knot-resolver/-/issues/195).
>>>>>>>>>>>> 
>>>>>>>>>>>> Initially, some of the lists have been *huge* which is why a simple HTTP download is not feasible. The porn list was over 100 MiB. We could have spent thousands on just traffic alone which I don’t have for this kind of project. It would also be unnecessary money being spent. There are simply better solutions out there. But then I built something that basically tests the data that we are receiving from upstream but simply checking if a listed domain still exists. The result was very astonishing to me.
>>>>>>>>>>>> 
>>>>>>>>>>>> So whenever someone adds a domain to the list, we will (eventually, but not immediately) check if we can resolve the domain’s SOA record. If not, we mark the domain as non-active and will no longer include them in the exported data. This brought down the porn list from just under 5 million domains to just 421k. On the sources page (https://www.ipfire.org/dnsbl/lists/porn/sources) I am listing the percentage of dead domains from each of them and the UT1 list has 94% dead domains. Wow.
>>>>>>>>>>>> 
>>>>>>>>>>>> If we cannot resolve the domain, neither can our users. So we would otherwise fill the lists with tons of domains that simply could never be reached. And if they cannot be reached, why would we block them? We would waste bandwidth and a lot of memory on each single client.
>>>>>>>>>>>> 
>>>>>>>>>>>> The other sources have similarly high rations of dead domains. Most of them are in the 50-80% range. Therefore I am happy that we are doing some extra work here to give our users much better data for their filtering.
>>>>>>>>>>> 
>>>>>>>>>>> Removing all dead entries sounds like an excellent step.
>>>>>>>>>>> 
>>>>>>>>>>> Regards,
>>>>>>>>>>> 
>>>>>>>>>>> Adolf.
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> So, if you like, please go and check out the RPZ blocking with Unbound. Instructions are on the page. I would be happy to hear how this is turning out.
>>>>>>>>>>>> 
>>>>>>>>>>>> Please let me know if there are any more questions, and I would be glad to answer them.
>>>>>>>>>>>> 
>>>>>>>>>>>> Happy New Year,
>>>>>>>>>>>> -Michael
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>> This email is just a brain dump from me to this list. I would be happy to answer any questions about implementation details, etc. if people are interested. Right now, this email is long enough already…
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> All the best,
>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -- 
>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -- 
>>>>>>>>> Sent from my laptop
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>> 
> <suricata-log.jpg><ipfire_dnsbl-violence.rules>

Re: Let's launch our own blocklists...

[Matthias Fischer]

On 26.01.2026 18:18, Michael Tremer wrote:
> Hello Matthias,

Hi Michael,

> Are you running this on Core Update 200?

No. On Core 199.

> There were some changes required so that we will extract the datasets from the rules tarballs. I am using a special feature in Suricata so that the lists won’t be using too much memory and will be quickly searchable:
> 
>   https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=f0b43241a501f7c545e3cb15f6989e945c60b3e2

Hm. I must have overlooked that.

Ok, I'll take a closer look at 'ids-functions.pl'. ;-)

> -Michael
> 
>> On 25 Jan 2026, at 17:50, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>> 
>> On 25.01.2026 15:40, Michael Tremer wrote:
>>> Hello Matthias,
>> 
>> Hi Michael,
>> 
>>> Nice catch!
>>> 
>>> I fixed it here and added the missing “;”:
>> 
>> Yep. The missing ";" seems to be fixed, but 'suricata' still doesn't
>> like our rules. I added 'violence' as an attachment... ;-)
>> 
>>>  https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=775561e322ceed43e255e5547bd76047b9f8a40b
>>> 
>>> If you go to the provider settings there is a button to force a ruleset update which should give you the fixed version. Please let me know if this works.
>> 
>> I did that - rules were updated, but no luck:
>> 
>> ***SNIP***
>> ...
>> 18:37:37  suricata:  [2577] <Info> -- Including configuration file
>> /var/ipfire/suricata/suricata-used-rulesfiles.yaml.
>> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
>> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop dns
>> any any -> any any (msg:"IPFire DBL [Violence] Blocked DNS Query";
>> dns.query; domain; dataset:isset,violence,type string,load
>> datasets/violence.txt; classtype:policy-violation; priority:2;
>> sid:1048577; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
>> metadata:dbl violence.dbl.ipfire.org;)" from file
>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 39
>> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
>> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop
>> http any any -> any any (msg:"IPFire DBL [Violence] Blocked HTTP
>> Request"; http.host; domain; dataset:isset,violence,type string,load
>> datasets/violence.txt; classtype:policy-violation; priority:2;
>> sid:1048578; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
>> metadata:dbl violence.dbl.ipfire.org;)" from file
>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 40
>> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
>> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop tls
>> any any -> any any (msg:"IPFire DBL [Violence] Blocked TLS Connection";
>> tls.sni; domain; dataset:isset,violence,type string,load
>> datasets/violence.txt; classtype:policy-violation; priority:2;
>> sid:1048579; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
>> metadata:dbl violence.dbl.ipfire.org;)" from file
>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 41
>> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
>> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop
>> quic any any -> any any (msg:"IPFire DBL [Violence] Blocked QUIC
>> Connection"; quic.sni; domain; dataset:isset,violence,type string,load
>> datasets/violence.txt; classtype:policy-violation; priority:2;
>> sid:1048580; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
>> metadata:dbl violence.dbl.ipfire.org;)" from file
>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 42
>> ...
>> ***SNAP***
>> 
>> For better reading - see attached screenshot.
>> 
>> Best
>> Matthias
>> 
>>> Best,
>>> -Michael
>>> 
>>>> On 24 Jan 2026, at 23:41, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>>>> 
>>>> On 23.01.2026 17:39, Michael Tremer wrote:
>>>>> Hello Matthias,
>>>> 
>>>> Hi Michael,
>>>> 
>>>>> Thank you very much for testing IPFire DBL.
>>>> 
>>>> No problem - I have news:
>>>> 
>>>> After taking a closer look to the IPS system logs, unfortunately I found
>>>> some parsing errors:
>>>> 
>>>> 'suricata' complains about missing ";".
>>>> 
>>>> ***SNIP***
>>>> ...
>>>> 00:32:40 suricata: [13343] <Info> -- Including configuration file
>>>> /var/ipfire/suricata/suricata-used-rulesfiles.yaml.
>>>> 00:32:40 suricata: [13343] <Error> -- no terminating ";" found
>>>> 00:32:40 suricata: [13343] <Error> -- error parsing signature "drop
>>>> dns any any -> any any (msg:"IPFire DBL [Advertising] Blocked DNS
>>>> Query"; dns.query; domain; dataset:isset,ads,type string,load
>>>> datasets/ads.txt; classtype:policy-violation; priority:3; sid:983041;
>>>> rev:1; reference:url,https://www.ipfire.org/dbl/ads; metadata:dbl
>>>> ads.dbl.ipfire.org)" from file /var/lib/suricata/ipfire_dnsbl-ads.rules
>>>> at line 72
>>>> 00:32:40 suricata: [13343] <Error> -- no terminating ";" found
>>>> ...
>>>> ***SNAP***
>>>> 
>>>> I tried, but didn't find the right place for any missing ";".
>>>> 
>>>> Can "anyone" confirm?
>>>> 
>>>> Best
>>>> Matthias
>>>> 
>>>>>> On 23 Jan 2026, at 15:02, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>>>>>> 
>>>>>> On 22.01.2026 12:33, Michael Tremer wrote:
>>>>>>> Hello everyone,
>>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> short feedback from me:
>>>>>> 
>>>>>> - I activated both the suricata (IPFire DBL - Domain Blocklist) - and
>>>>>> the URLfilter lists from 'dbl.ipfire.org'.
>>>>> 
>>>>> This is an interesting case. What I didn’t manage to test yet is what happens when Suricata blocks the connection first. If URL Filter sees a domain that is being blocked it will either send you an error page if you are using HTTP, or simply close the connection if it is HTTPS. However, when Suricata comes first in the chain (and it will), it might close the connection because URL Filter has received the request. In the case of HTTPS this does not make any difference because the connection will be closed, but in the HTTP case you won’t see an error page any more and instead have the connection closed, too. You are basically losing the explicit error notification which is a little bit annoying.
>>>>> 
>>>>> We could have the same when we are doing the same with Unbound and DNS filtering. Potentially we would need to whitelist the local DNS resolver then, but how is Suricata supposed to know that the same categories are activated in both places?
>>>>> 
>>>>>> - I even took the 'smart-tv' domains from the IFire DBL blacklist and
>>>>>> copied/pasted them in my fritzbox filter lists.
>>>>> 
>>>>> LOL Why not use IPFire to filter this as well?
>>>>> 
>>>>>> Everything works as expected. Besides, the download of the IPFire
>>>>>> DBL-list loads a lot faster than the list from 'Univ. Toulouse'... ;-)
>>>>> 
>>>>> Yes, we don’t have much traffic on the server, yet.
>>>>> 
>>>>>> Functionality is good - no false positives or seen problems. Good work -
>>>>>> thanks!
>>>>> 
>>>>> Nice. We need to distinguish a little between what is a technical issue and what is a false-positive/missing domain on the list. However, testing both at the same time is something we will all cope quite well with :)
>>>>> 
>>>>> -Michael
>>>>> 
>>>>>> Best
>>>>>> Matthias
>>>>>> 
>>>>>>> Over the past few weeks I have made significant progress on this all, and I think we're getting close to something the community will be really happy with. I'd love to get feedback from the team before we finalise things.
>>>>>>> 
>>>>>>> So what has happened?
>>>>>>> 
>>>>>>> First of all, the entire project has been renamed. DNSBL is not entirely what this is. Although the lists can be thrown into DNS, they have much more use outside of it that I thought we should simply go with DBL, short for Domain Blocklist. After all, we are only importing domains. The new home of the project therefore is https://www.ipfire.org/dbl
>>>>>>> 
>>>>>>> I have added a couple more lists that I thought interesting and I have added a couple more sources that I considered a good start. Hopefully, we will soon gather some more feedback on how well this is all holding up. My main focus has however been on the technology that will power this project.
>>>>>>> 
>>>>>>> One of the bigger challenges was to create Suricata rules from the lists. Initially I tried to create a ton of rules but since our lists are so large, this quickly became too complicated. I have now settled on using a feature that is only available in more recent versions of Suricata (I believe 7 and later), but since we are already on Suricata 8 in IPFire this won’t be a problem for us. All domains for each list are basically compiled into one massively large dataset and one single rule is referring to that dataset. This way, we won’t have the option to remove any false-positives, but at least Suricata and the GUI won’t starve a really bad death when loading millions of rules.
>>>>>>> 
>>>>>>> Suricata will now be able to use our rules to block access to any listed domains of each of the categories over DNS, HTTP, TLS or QUIC. Although I don’t expect many users to use Suricata to block porn or other things, this is a great backstop to enforce any policy like that. For example, if there is a user on the network who is trying to circumvent the DNS server that might filter out certain domains, even after getting an IP address resolved through other means, they won’t be able to open a TLS/QUIC connection or send a HTTP request to all blocked domains. Some people have said they were interested in blocking DNS-over-HTTPS and this is a perfect way to do this and actually be sure that any server that is being blocked on the list will actually be completely inaccessible.
>>>>>>> 
>>>>>>> Those Suricata rules are already available for testing in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=9eb8751487d23dd354a105c28bdbbb0398fe6e85
>>>>>>> 
>>>>>>> I have chosen various severities for the lists. If someone was to block advertising using DBL, this is fine, but not a very severe alert. If someone chooses to block malware and there is a system on the network trying to access those domains, this is an alert worth being investigated by an admin. Our new Suricata Reporter will show those violations in different colours based on the severity which helps to identify the right alerts to further investigate.
>>>>>>> 
>>>>>>> Formerly I have asked you to test the lists using URL Filter. Those rules are now available as well in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=db160694279a4b10378447f775dd536fdfcfb02a
>>>>>>> 
>>>>>>> I talked about a method to remove any dead domains from any sources which is a great way to keep our lists smaller. The pure size of them is a problem in so many ways. That check was however a little bit too ambitious and I had to make it a little bit less eager. Basically if we are in doubt, we need to still list the domain because it might be resolvable by a user.
>>>>>>> 
>>>>>>> https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=bb5b6e33b731501d45dea293505f7d42a61d5ce7
>>>>>>> 
>>>>>>> So how else could we make the lists smaller without losing any actual data? Since we sometimes list a whole TLD (e.g. .xxx or .porn), there is very little point in listing any domains of this TLD. They will always be caught anyways. So I built a check that marks all domains that don’t need to be included on the exported lists because they will never be needed and was able to shrink the size of the lists by a lot again.
>>>>>>> 
>>>>>>> The website does not show this data, but the API returns the number of “subsumed” domains (I didn’t have a better name):
>>>>>>> 
>>>>>>> curl https://api.dbl.ipfire.org/lists | jq .
>>>>>>> 
>>>>>>> The number shown would normally be added to the total number of domains and usually cuts the size of the list by 50-200%.
>>>>>>> 
>>>>>>> Those stats will now also be stored in a history table so that we will be able to track growth of all lists.
>>>>>>> 
>>>>>>> Furthermore, the application will now send email notifications for any incoming reports. This way, we will be able to stay in close touch with the reporters and keep them up to date on their submissions as well as inform moderators that there is something to have a look at.
>>>>>>> 
>>>>>>> The search has been refactored as well, so that we can show clearly whether something is blocked or not at one glance: https://www.ipfire.org/dbl/search?q=github.com. There is detailed information available on all domains and what happened to them. In case of GitHub.com, this seems to be blocked and unblocked by someone all of the time and we can see a clear audit trail of that: https://www.ipfire.org/dbl/lists/malware/domains/github.com
>>>>>>> 
>>>>>>> On the DNS front, I have added some metadata to the zones so that people can programmatically request some data, like when it has been last updated (in a human-friendly timestamp and not only the serial), license, description and so on:
>>>>>>> 
>>>>>>> # dig +short ANY _info.ads.dbl.ipfire.org @primary.dbl.ipfire.org
>>>>>>> "total-domains=42226"
>>>>>>> "license=CC BY-SA 4.0"
>>>>>>> "updated-at=2026-01-20T22:17:02.409933+00:00"
>>>>>>> "description=Blocks domains used for ads, tracking, and ad delivery”
>>>>>>> 
>>>>>>> Now, I would like to hear more feedback from you. I know we've all been stretched thin lately, so I especially appreciate anyone who has time to review and provide input. Ideas, just say if you like it or not. Where this could go in the future?
>>>>>>> 
>>>>>>> Looking ahead, I would like us to start thinking about the RPZ feature that has been on the wishlist. IPFire DBL has been a bigger piece of work, and I think it's worth having a conversation about sustainability. Resources for this need to be allocated and paid for. Open source is about freedom, not free beer — and to keep building features like this, we will need to explore some funding options. I would be interested to hear any ideas you might have that could work for IPFire.
>>>>>>> 
>>>>>>> Please share your thoughts on the mailing list when you can — even a quick 'looks good' or 'I have concerns about X' is valuable. Public discussion helps everyone stay in the loop and contribute.
>>>>>>> 
>>>>>>> I am aiming to move forward with this in a week's time, so if you have input, now would be a good time to share it.
>>>>>>> 
>>>>>>> Best,
>>>>>>> -Michael
>>>>>>> 
>>>>>>>> On 6 Jan 2026, at 10:20, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>>>>>>> 
>>>>>>>> Good Morning Adolf,
>>>>>>>> 
>>>>>>>> I had a look at this problem yesterday and it seems that parsing the format is becoming a little bit difficult this way. Since this is only affecting very few domains, I have simply whitelisted them all manually and duckduckgo.com <http://duckduckgo.com/> and others should now be easily reachable again.
>>>>>>>> 
>>>>>>>> Please let me know if you have any more findings.
>>>>>>>> 
>>>>>>>> All the best,
>>>>>>>> -Michael
>>>>>>>> 
>>>>>>>>> On 5 Jan 2026, at 11:48, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>>>>>>>> 
>>>>>>>>> Hello Adolf,
>>>>>>>>> 
>>>>>>>>> This is a good find.
>>>>>>>>> 
>>>>>>>>> But if duckduckgo.com <http://duckduckgo.com/> is blocked, we will have to have a source somewhere that blocks that domain. Not only a sub-domain of it. Otherwise we have a bug somewhere.
>>>>>>>>> 
>>>>>>>>> This is most likely as the domain is listed here, but with some stuff afterwards:
>>>>>>>>> 
>>>>>>>>> https://raw.githubusercontent.com/mtxadmin/ublock/refs/heads/master/hosts/_malware_typo
>>>>>>>>> 
>>>>>>>>> We strip everything after a # away because we consider it a comment. However, that causes that there is only a line with the domain left which will cause it being listed.
>>>>>>>>> 
>>>>>>>>> The # sign is used as some special character but at the same time it is being used for comments.
>>>>>>>>> 
>>>>>>>>> I will fix this and then refresh the list.
>>>>>>>>> 
>>>>>>>>> -Michael
>>>>>>>>> 
>>>>>>>>>> On 5 Jan 2026, at 11:31, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi Michael,
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On 05/01/2026 12:11, Adolf Belka wrote:
>>>>>>>>>>> Hi Michael,
>>>>>>>>>>> 
>>>>>>>>>>> I have found that the malware list includes duckduckgo.com
>>>>>>>>>>> 
>>>>>>>>>> I have checked through the various sources used for the malware list.
>>>>>>>>>> 
>>>>>>>>>> The ShadowWhisperer (Tracking) list has improving.duckduckgo.com in its list. I suspect that this one is the one causing the problem.
>>>>>>>>>> 
>>>>>>>>>> The mtxadmin (_malware_typo) list has duckduckgo.com mentioned 3 times but not directly as a domain name - looks more like a reference.
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> 
>>>>>>>>>> Adolf.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> Regards,
>>>>>>>>>>> Adolf.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On 02/01/2026 14:02, Adolf Belka wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> 
>>>>>>>>>>>> On 02/01/2026 12:09, Michael Tremer wrote:
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 30 Dec 2025, at 14:05, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 29/12/2025 13:05, Michael Tremer wrote:
>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I hope everyone had a great Christmas and a couple of quiet days to relax from all the stress that was the year 2025.
>>>>>>>>>>>>>> Still relaxing.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Very good, so let’s have a strong start into 2026 now!
>>>>>>>>>>>> 
>>>>>>>>>>>> Starting next week, yes.
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Having a couple of quieter days, I have been working on a new, little (hopefully) side project that has probably been high up on our radar since the Shalla list has shut down in 2020, or maybe even earlier. The goal of the project is to provide good lists with categories of domain names which are usually used to block access to these domains.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> How did we get here?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> As stated before, the URL filter feature in IPFire has the problem that there are not many good blocklists available any more. There used to be a couple more - most famously the Shalla list - but we are now down to a single list from the University of Toulouse. It is a great list, but it is not always the best fit for all users.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Then there has been talk about whether we could implement more blocking features into IPFire that don’t involve the proxy. Most famously blocking over DNS. The problem here remains a the blocking feature is only as good as the data that is fed into it. Some people have been putting forward a number of lists that were suitable for them, but they would not have replaced the blocking functionality as we know it. Their aim is to provide “one list for everything” but that is not what people usually want. It is targeted at a classic home user and the only separation that is being made is any adult/porn/NSFW content which usually is put into a separate list.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> It would have been technically possible to include these lists and let the users decide, but that is not the aim of IPFire. We want to do the job for the user so that their job is getting easier. Including obscure lists that don’t have a clear outline of what they actually want to block (“bad content” is not a category) and passing the burden of figuring out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with cream on top is really not going to work. It is all confusing and will lead to a bad user experience.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> An even bigger problem that is however completely impossible to solve is bad licensing of these lists. A user has asked the publisher of the HaGeZi list whether they could be included in IPFire and under what terms. The response was that the list is available under the terms of the GNU General Public License v3, but that does not seem to be true. The list contains data from various sources. Many of them are licensed under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public agreement that this data may be redistributed, there is a huge legal issue here. We would expose our users to potential copyright infringement which we cannot do under any circumstances. Furthermore many lists are available under a non-commercial license which excludes them from being used in any kind of business. Plenty of IPFire systems are running in businesses, if not even the vast majority.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> In short, these lists are completely unusable for us. Apart from HaGeZi, I consider OISD to have the same problem.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Enough about all the things that are bad. Let’s talk about the new, good things:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Many blacklists on the internet are an amalgamation of other lists. These lists vary in quality with some of them being not that good and without a clear focus and others being excellent data. Since we don’t have the man power to start from scratch, I felt that we can copy the concept that HaGeZi and OISD have started and simply create a new list that is based on other lists at the beginning to have a good starting point. That way, we have much better control over what is going on these lists and we can shape and mould them as we need them. Most importantly, we don’t create a single lists, but many lists that have a clear focus and allow users to choose what they want to block and what not.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> So the current experimental stage that I am in has these lists:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> * Ads
>>>>>>>>>>>>>>> * Dating
>>>>>>>>>>>>>>> * DoH
>>>>>>>>>>>>>>> * Gambling
>>>>>>>>>>>>>>> * Malware
>>>>>>>>>>>>>>> * Porn
>>>>>>>>>>>>>>> * Social
>>>>>>>>>>>>>>> * Violence
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The categories have been determined by what source lists we have available with good data and are compatible with our chosen license CC BY-SA 4.0. This is the same license that we are using for the IPFire Location database, too.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The main use-cases for any kind of blocking are to comply with legal requirements in networks with children (i.e. schools) to remove any kind of pornographic content, sometimes block social media as well. Gambling and violence are commonly blocked, too. Even more common would be filtering advertising and any malicious content.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The latter is especially difficult because so many source lists throw phishing, spyware, malvertising, tracking and other things into the same bucket. Here this is currently all in the malware list which has therefore become quite large. I am not sure whether this will stay like this in the future or if we will have to make some adjustments, but that is exactly why this is now entering some larger testing.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> What has been built so far? In order to put these lists together properly, track any data about where it is coming from, I have built a tool in Python available here:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> https://git.ipfire.org/?p=dnsbl.git;a=summary
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> This tool will automatically update all lists once an hour if there have been any changes and export them in various formats. The exported lists are available for download here:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> https://dnsbl.ipfire.org/lists/
>>>>>>>>>>>>>> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the custom url works fine.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> However you need to remember not to put the https:// at the front of the url otherwise the WUI page completes without any error messages but leaves an error message in the system logs saying
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> URL filter blacklist - ERROR: Not a valid URL filter blacklist
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I found this out the hard way.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Oh yes, I forgot that there is a field on the web UI. If that does not accept https:// as a prefix, please file a bug and we will fix it.
>>>>>>>>>>>> 
>>>>>>>>>>>> I will confirm it and raise a bug.
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The other thing I noticed is that if you already have the Toulouse University list downloaded and you then change to the ipfire custom url then all the existing Toulouse blocklists stay in the directory on IPFire and so you end up with a huge number of category tick boxes, most of which are the old Toulouse ones, which are still available to select and it is not clear which ones are from Toulouse and which ones from IPFire.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Yes, I got the same thing, too. I think this is a bug, too, because otherwise you would have a lot of unused categories lying around that will never be updated. You cannot even tell which ones are from the current list and which ones from the old list.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Long-term we could even consider to remove the Univ. Toulouse list entirely and only have our own lists available which would make the problem go away.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I think if the blocklist URL source is changed or a custom url is provided the first step should be to remove the old ones already existing.
>>>>>>>>>>>>>> That might be a problem because users can also create their own blocklists and I believe those go into the same directory.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Good thought. We of course cannot delete the custom lists.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Without clearing out the old blocklists you end up with a huge number of checkboxes for lists but it is not clear what happens if there is a category that has the same name for the Toulouse list and the IPFire list such as gambling. I will have a look at that and see what happens.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Not sure what the best approach to this is.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I believe it is removing all old content.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Manually deleting all contents of the urlfilter/blacklists/ directory and then selecting the IPFire blocklist url for the custom url I end up with only the 8 categories from the IPFire list.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I have tested some gambling sites from the IPFire list and the block worked on some. On others the site no longer exists so there is nothing to block or has been changed to an https site and in that case it went straight through. Also if I chose the http version of the link, it was automatically changed to https and went through without being blocked.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The entire IPFire infrastructure always requires HTTPS. If you start using HTTP, you will be automatically redirected. It is 2026 and we don’t need to talk HTTP any more :)
>>>>>>>>>>>> 
>>>>>>>>>>>> Some of the domains in the gambling list (maybe quite a lot) seem to only have an http access. If I tried https it came back with the fact that it couldn't find it.
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I am glad to hear that the list is actually blocking. It would have been bad if it didn’t. Now we have the big task to check out the “quality” - however that can be determined. I think this is what needs some time…
>>>>>>>>>>>>> 
>>>>>>>>>>>>> In the meantime I have set up a small page on our website:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> https://www.ipfire.org/dnsbl
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I would like to run this as a first-class project inside IPFire like we are doing with IPFire Location. That means that we need to tell people about what we are doing. Hopefully this page is a little start.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Initially it has a couple of high-level bullet points about what we are trying to achieve. I don’t think the text is very good, yet, but it is the best I had in that moment. There is then also a list of the lists that we currently offer. For each list, a detailed page will tell you about the license, how many domains are listed, when the last update has been, the sources and even there is a history page that shows all the changes whenever they have happened.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Finally there is a section that explains “How To Use?” the list which I would love to extend to include AdGuard Plus and things like that as well as Pi-Hole and whatever else could use the list. In a later step we should go ahead and talk to any projects to include our list(s) into their dropdown so that people can enable them nice and easy.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Behind the web page there is an API service that is running on the host that is running the DNSBL. The frontend web app that is running www.ipfire.org <http://www.ipfire.org/> is connecting to that API service to fetch the current lists, any details and so on. That way, we can split the logic and avoid creating a huge monolith of a web app. This also means that page could be down a little as I am still working on the entire thing and will frequently restart it.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The API documentation is available here and the API is publicly available: https://api.dnsbl.ipfire.org/docs
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The website/API allows to file reports for anything that does not seem to be right on any of the lists. I would like to keep it as an open process, however, long-term, this cannot cost us any time. In the current stage, the reports are getting filed and that is about it. I still need to build out some way for admins or moderators (I am not sure what kind of roles I want to have here) to accept or reject those reports.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> In case of us receiving a domain from a source list, I would rather like to submit a report to upstream for them to de-list. That way, we don’t have any admin to do and we are contributing back to other list. That would be a very good thing to do. We cannot however throw tons of emails at some random upstream projects without co-ordinating this first. By not reporting upstream, we will probably over time create large whitelists and I am not sure if that is a good thing to do.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Finally, there is a search box that can be used to find out if a domain is listed on any of the lists.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> If you download and open any of the files, you will see a large header that includes copyright information and lists all sources that have been used to create the individual lists. This way we ensure maximum transparency, comply with the terms of the individual licenses of the source lists and give credit to the people who help us to put together the most perfect list for our users.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I would like this to become a project that is not only being used in IPFire. We can and will be compatible with other solutions like AdGuard, PiHole so that people can use our lists if they would like to even though they are not using IPFire. Hopefully, these users will also feed back to us so that we can improve our lists over time and make them one of the best options out there.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> All lists are available as a simple text file that lists the domains. Then there is a hosts file available as well as a DNS zone file and an RPZ file. Each list is individually available to be used in squidGuard and there is a larger tarball available with all lists that can be used in IPFire’s URL Filter. I am planning to add Suricata/Snort signatures whenever I have time to do so. Even though it is not a good idea to filter pornographic content this way, I suppose that catching malware and blocking DoH are good use-cases for an IPS. Time will tell…
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> As a start, we will make these lists available in IPFire’s URL Filter and collect some feedback about how we are doing. Afterwards, we can see where else we can take this project.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> If you want to enable this on your system, simply add the URL to your autoupdate.urls file like here:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f
>>>>>>>>>>>>>> I also tested out adding the IPFire url to autoupdate.urls and that also worked fine for me.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Very good. Should we include this already with Core Update 200? I don’t think we would break anything, but we might already gain a couple more people who are helping us to test this all?
>>>>>>>>>>>> 
>>>>>>>>>>>> I think that would be a good idea.
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The next step would be to build and test our DNS infrastructure. In the “How To Use?” Section on the pages of the individual lists, you can already see some instructions on how to use the lists as an RPZ. In comparison to other “providers”, I would prefer if people would be using DNS to fetch the lists. This is simply to push out updates in a cheap way for us and also do it very regularly.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Initially, clients will pull the entire list using AXFR. There is no way around this as they need to have the data in the first place. After that, clients will only need the changes. As you can see in the history, the lists don’t actually change that often. Sometimes only once a day and therefore downloading the entire list again would be a huge waste of data, both on the client side, but also for us hosting then.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Some other providers update their lists “every 10 minutes”, and there won't be any changes whatsoever. We don’t do that. We will only export the lists again when they have actually changed. The timestamps on the files that we offer using HTTPS can be checked by clients so that they won’t re-download the list again if it has not been changed. But using HTTPS still means that we would have to re-download the entire list and not only the changes.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Using DNS and IXFR will update the lists by only transferring a few kilobytes and therefore we can have clients check once an hour if a list has actually changed and only send out the raw changes. That way, we will be able to serve millions of clients at very cheap cost and they will always have a very up to date list.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> As far as I can see any DNS software that supports RPZs supports AXFR/IXFR with exception of Knot Resolver which expects the zone to be downloaded externally. There is a ticket for AXFR/IXFR support (https://gitlab.nic.cz/knot/knot-resolver/-/issues/195).
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Initially, some of the lists have been *huge* which is why a simple HTTP download is not feasible. The porn list was over 100 MiB. We could have spent thousands on just traffic alone which I don’t have for this kind of project. It would also be unnecessary money being spent. There are simply better solutions out there. But then I built something that basically tests the data that we are receiving from upstream but simply checking if a listed domain still exists. The result was very astonishing to me.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> So whenever someone adds a domain to the list, we will (eventually, but not immediately) check if we can resolve the domain’s SOA record. If not, we mark the domain as non-active and will no longer include them in the exported data. This brought down the porn list from just under 5 million domains to just 421k. On the sources page (https://www.ipfire.org/dnsbl/lists/porn/sources) I am listing the percentage of dead domains from each of them and the UT1 list has 94% dead domains. Wow.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> If we cannot resolve the domain, neither can our users. So we would otherwise fill the lists with tons of domains that simply could never be reached. And if they cannot be reached, why would we block them? We would waste bandwidth and a lot of memory on each single client.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The other sources have similarly high rations of dead domains. Most of them are in the 50-80% range. Therefore I am happy that we are doing some extra work here to give our users much better data for their filtering.
>>>>>>>>>>>> 
>>>>>>>>>>>> Removing all dead entries sounds like an excellent step.
>>>>>>>>>>>> 
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> 
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> So, if you like, please go and check out the RPZ blocking with Unbound. Instructions are on the page. I would be happy to hear how this is turning out.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Please let me know if there are any more questions, and I would be glad to answer them.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Happy New Year,
>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>> This email is just a brain dump from me to this list. I would be happy to answer any questions about implementation details, etc. if people are interested. Right now, this email is long enough already…
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> All the best,
>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> -- 
>>>>>>>>>> Sent from my laptop
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>> <suricata-log.jpg><ipfire_dnsbl-violence.rules>
> 

Re: Let's launch our own blocklists...

[Matthias Fischer]

On 26.01.2026 18:18, Michael Tremer wrote:
> Hello Matthias,

Hi,

> Are you running this on Core Update 200?

Now running on Core 199 - after applying...

> There were some changes required so that we will extract the datasets from the rules tarballs. I am using a special feature in Suricata so that the lists won’t be using too much memory and will be quickly searchable:
> 
>   https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=f0b43241a501f7c545e3cb15f6989e945c60b3e2

...these patches.

No more errors. ;-)

Best
Matthias

> -Michael
> 
>> On 25 Jan 2026, at 17:50, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>> 
>> On 25.01.2026 15:40, Michael Tremer wrote:
>>> Hello Matthias,
>> 
>> Hi Michael,
>> 
>>> Nice catch!
>>> 
>>> I fixed it here and added the missing “;”:
>> 
>> Yep. The missing ";" seems to be fixed, but 'suricata' still doesn't
>> like our rules. I added 'violence' as an attachment... ;-)
>> 
>>>  https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=775561e322ceed43e255e5547bd76047b9f8a40b
>>> 
>>> If you go to the provider settings there is a button to force a ruleset update which should give you the fixed version. Please let me know if this works.
>> 
>> I did that - rules were updated, but no luck:
>> 
>> ***SNIP***
>> ...
>> 18:37:37  suricata:  [2577] <Info> -- Including configuration file
>> /var/ipfire/suricata/suricata-used-rulesfiles.yaml.
>> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
>> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop dns
>> any any -> any any (msg:"IPFire DBL [Violence] Blocked DNS Query";
>> dns.query; domain; dataset:isset,violence,type string,load
>> datasets/violence.txt; classtype:policy-violation; priority:2;
>> sid:1048577; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
>> metadata:dbl violence.dbl.ipfire.org;)" from file
>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 39
>> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
>> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop
>> http any any -> any any (msg:"IPFire DBL [Violence] Blocked HTTP
>> Request"; http.host; domain; dataset:isset,violence,type string,load
>> datasets/violence.txt; classtype:policy-violation; priority:2;
>> sid:1048578; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
>> metadata:dbl violence.dbl.ipfire.org;)" from file
>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 40
>> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
>> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop tls
>> any any -> any any (msg:"IPFire DBL [Violence] Blocked TLS Connection";
>> tls.sni; domain; dataset:isset,violence,type string,load
>> datasets/violence.txt; classtype:policy-violation; priority:2;
>> sid:1048579; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
>> metadata:dbl violence.dbl.ipfire.org;)" from file
>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 41
>> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
>> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop
>> quic any any -> any any (msg:"IPFire DBL [Violence] Blocked QUIC
>> Connection"; quic.sni; domain; dataset:isset,violence,type string,load
>> datasets/violence.txt; classtype:policy-violation; priority:2;
>> sid:1048580; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
>> metadata:dbl violence.dbl.ipfire.org;)" from file
>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 42
>> ...
>> ***SNAP***
>> 
>> For better reading - see attached screenshot.
>> 
>> Best
>> Matthias
>> 
>>> Best,
>>> -Michael
>>> 
>>>> On 24 Jan 2026, at 23:41, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>>>> 
>>>> On 23.01.2026 17:39, Michael Tremer wrote:
>>>>> Hello Matthias,
>>>> 
>>>> Hi Michael,
>>>> 
>>>>> Thank you very much for testing IPFire DBL.
>>>> 
>>>> No problem - I have news:
>>>> 
>>>> After taking a closer look to the IPS system logs, unfortunately I found
>>>> some parsing errors:
>>>> 
>>>> 'suricata' complains about missing ";".
>>>> 
>>>> ***SNIP***
>>>> ...
>>>> 00:32:40 suricata: [13343] <Info> -- Including configuration file
>>>> /var/ipfire/suricata/suricata-used-rulesfiles.yaml.
>>>> 00:32:40 suricata: [13343] <Error> -- no terminating ";" found
>>>> 00:32:40 suricata: [13343] <Error> -- error parsing signature "drop
>>>> dns any any -> any any (msg:"IPFire DBL [Advertising] Blocked DNS
>>>> Query"; dns.query; domain; dataset:isset,ads,type string,load
>>>> datasets/ads.txt; classtype:policy-violation; priority:3; sid:983041;
>>>> rev:1; reference:url,https://www.ipfire.org/dbl/ads; metadata:dbl
>>>> ads.dbl.ipfire.org)" from file /var/lib/suricata/ipfire_dnsbl-ads.rules
>>>> at line 72
>>>> 00:32:40 suricata: [13343] <Error> -- no terminating ";" found
>>>> ...
>>>> ***SNAP***
>>>> 
>>>> I tried, but didn't find the right place for any missing ";".
>>>> 
>>>> Can "anyone" confirm?
>>>> 
>>>> Best
>>>> Matthias
>>>> 
>>>>>> On 23 Jan 2026, at 15:02, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>>>>>> 
>>>>>> On 22.01.2026 12:33, Michael Tremer wrote:
>>>>>>> Hello everyone,
>>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> short feedback from me:
>>>>>> 
>>>>>> - I activated both the suricata (IPFire DBL - Domain Blocklist) - and
>>>>>> the URLfilter lists from 'dbl.ipfire.org'.
>>>>> 
>>>>> This is an interesting case. What I didn’t manage to test yet is what happens when Suricata blocks the connection first. If URL Filter sees a domain that is being blocked it will either send you an error page if you are using HTTP, or simply close the connection if it is HTTPS. However, when Suricata comes first in the chain (and it will), it might close the connection because URL Filter has received the request. In the case of HTTPS this does not make any difference because the connection will be closed, but in the HTTP case you won’t see an error page any more and instead have the connection closed, too. You are basically losing the explicit error notification which is a little bit annoying.
>>>>> 
>>>>> We could have the same when we are doing the same with Unbound and DNS filtering. Potentially we would need to whitelist the local DNS resolver then, but how is Suricata supposed to know that the same categories are activated in both places?
>>>>> 
>>>>>> - I even took the 'smart-tv' domains from the IFire DBL blacklist and
>>>>>> copied/pasted them in my fritzbox filter lists.
>>>>> 
>>>>> LOL Why not use IPFire to filter this as well?
>>>>> 
>>>>>> Everything works as expected. Besides, the download of the IPFire
>>>>>> DBL-list loads a lot faster than the list from 'Univ. Toulouse'... ;-)
>>>>> 
>>>>> Yes, we don’t have much traffic on the server, yet.
>>>>> 
>>>>>> Functionality is good - no false positives or seen problems. Good work -
>>>>>> thanks!
>>>>> 
>>>>> Nice. We need to distinguish a little between what is a technical issue and what is a false-positive/missing domain on the list. However, testing both at the same time is something we will all cope quite well with :)
>>>>> 
>>>>> -Michael
>>>>> 
>>>>>> Best
>>>>>> Matthias
>>>>>> 
>>>>>>> Over the past few weeks I have made significant progress on this all, and I think we're getting close to something the community will be really happy with. I'd love to get feedback from the team before we finalise things.
>>>>>>> 
>>>>>>> So what has happened?
>>>>>>> 
>>>>>>> First of all, the entire project has been renamed. DNSBL is not entirely what this is. Although the lists can be thrown into DNS, they have much more use outside of it that I thought we should simply go with DBL, short for Domain Blocklist. After all, we are only importing domains. The new home of the project therefore is https://www.ipfire.org/dbl
>>>>>>> 
>>>>>>> I have added a couple more lists that I thought interesting and I have added a couple more sources that I considered a good start. Hopefully, we will soon gather some more feedback on how well this is all holding up. My main focus has however been on the technology that will power this project.
>>>>>>> 
>>>>>>> One of the bigger challenges was to create Suricata rules from the lists. Initially I tried to create a ton of rules but since our lists are so large, this quickly became too complicated. I have now settled on using a feature that is only available in more recent versions of Suricata (I believe 7 and later), but since we are already on Suricata 8 in IPFire this won’t be a problem for us. All domains for each list are basically compiled into one massively large dataset and one single rule is referring to that dataset. This way, we won’t have the option to remove any false-positives, but at least Suricata and the GUI won’t starve a really bad death when loading millions of rules.
>>>>>>> 
>>>>>>> Suricata will now be able to use our rules to block access to any listed domains of each of the categories over DNS, HTTP, TLS or QUIC. Although I don’t expect many users to use Suricata to block porn or other things, this is a great backstop to enforce any policy like that. For example, if there is a user on the network who is trying to circumvent the DNS server that might filter out certain domains, even after getting an IP address resolved through other means, they won’t be able to open a TLS/QUIC connection or send a HTTP request to all blocked domains. Some people have said they were interested in blocking DNS-over-HTTPS and this is a perfect way to do this and actually be sure that any server that is being blocked on the list will actually be completely inaccessible.
>>>>>>> 
>>>>>>> Those Suricata rules are already available for testing in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=9eb8751487d23dd354a105c28bdbbb0398fe6e85
>>>>>>> 
>>>>>>> I have chosen various severities for the lists. If someone was to block advertising using DBL, this is fine, but not a very severe alert. If someone chooses to block malware and there is a system on the network trying to access those domains, this is an alert worth being investigated by an admin. Our new Suricata Reporter will show those violations in different colours based on the severity which helps to identify the right alerts to further investigate.
>>>>>>> 
>>>>>>> Formerly I have asked you to test the lists using URL Filter. Those rules are now available as well in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=db160694279a4b10378447f775dd536fdfcfb02a
>>>>>>> 
>>>>>>> I talked about a method to remove any dead domains from any sources which is a great way to keep our lists smaller. The pure size of them is a problem in so many ways. That check was however a little bit too ambitious and I had to make it a little bit less eager. Basically if we are in doubt, we need to still list the domain because it might be resolvable by a user.
>>>>>>> 
>>>>>>> https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=bb5b6e33b731501d45dea293505f7d42a61d5ce7
>>>>>>> 
>>>>>>> So how else could we make the lists smaller without losing any actual data? Since we sometimes list a whole TLD (e.g. .xxx or .porn), there is very little point in listing any domains of this TLD. They will always be caught anyways. So I built a check that marks all domains that don’t need to be included on the exported lists because they will never be needed and was able to shrink the size of the lists by a lot again.
>>>>>>> 
>>>>>>> The website does not show this data, but the API returns the number of “subsumed” domains (I didn’t have a better name):
>>>>>>> 
>>>>>>> curl https://api.dbl.ipfire.org/lists | jq .
>>>>>>> 
>>>>>>> The number shown would normally be added to the total number of domains and usually cuts the size of the list by 50-200%.
>>>>>>> 
>>>>>>> Those stats will now also be stored in a history table so that we will be able to track growth of all lists.
>>>>>>> 
>>>>>>> Furthermore, the application will now send email notifications for any incoming reports. This way, we will be able to stay in close touch with the reporters and keep them up to date on their submissions as well as inform moderators that there is something to have a look at.
>>>>>>> 
>>>>>>> The search has been refactored as well, so that we can show clearly whether something is blocked or not at one glance: https://www.ipfire.org/dbl/search?q=github.com. There is detailed information available on all domains and what happened to them. In case of GitHub.com, this seems to be blocked and unblocked by someone all of the time and we can see a clear audit trail of that: https://www.ipfire.org/dbl/lists/malware/domains/github.com
>>>>>>> 
>>>>>>> On the DNS front, I have added some metadata to the zones so that people can programmatically request some data, like when it has been last updated (in a human-friendly timestamp and not only the serial), license, description and so on:
>>>>>>> 
>>>>>>> # dig +short ANY _info.ads.dbl.ipfire.org @primary.dbl.ipfire.org
>>>>>>> "total-domains=42226"
>>>>>>> "license=CC BY-SA 4.0"
>>>>>>> "updated-at=2026-01-20T22:17:02.409933+00:00"
>>>>>>> "description=Blocks domains used for ads, tracking, and ad delivery”
>>>>>>> 
>>>>>>> Now, I would like to hear more feedback from you. I know we've all been stretched thin lately, so I especially appreciate anyone who has time to review and provide input. Ideas, just say if you like it or not. Where this could go in the future?
>>>>>>> 
>>>>>>> Looking ahead, I would like us to start thinking about the RPZ feature that has been on the wishlist. IPFire DBL has been a bigger piece of work, and I think it's worth having a conversation about sustainability. Resources for this need to be allocated and paid for. Open source is about freedom, not free beer — and to keep building features like this, we will need to explore some funding options. I would be interested to hear any ideas you might have that could work for IPFire.
>>>>>>> 
>>>>>>> Please share your thoughts on the mailing list when you can — even a quick 'looks good' or 'I have concerns about X' is valuable. Public discussion helps everyone stay in the loop and contribute.
>>>>>>> 
>>>>>>> I am aiming to move forward with this in a week's time, so if you have input, now would be a good time to share it.
>>>>>>> 
>>>>>>> Best,
>>>>>>> -Michael
>>>>>>> 
>>>>>>>> On 6 Jan 2026, at 10:20, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>>>>>>> 
>>>>>>>> Good Morning Adolf,
>>>>>>>> 
>>>>>>>> I had a look at this problem yesterday and it seems that parsing the format is becoming a little bit difficult this way. Since this is only affecting very few domains, I have simply whitelisted them all manually and duckduckgo.com <http://duckduckgo.com/> and others should now be easily reachable again.
>>>>>>>> 
>>>>>>>> Please let me know if you have any more findings.
>>>>>>>> 
>>>>>>>> All the best,
>>>>>>>> -Michael
>>>>>>>> 
>>>>>>>>> On 5 Jan 2026, at 11:48, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>>>>>>>> 
>>>>>>>>> Hello Adolf,
>>>>>>>>> 
>>>>>>>>> This is a good find.
>>>>>>>>> 
>>>>>>>>> But if duckduckgo.com <http://duckduckgo.com/> is blocked, we will have to have a source somewhere that blocks that domain. Not only a sub-domain of it. Otherwise we have a bug somewhere.
>>>>>>>>> 
>>>>>>>>> This is most likely as the domain is listed here, but with some stuff afterwards:
>>>>>>>>> 
>>>>>>>>> https://raw.githubusercontent.com/mtxadmin/ublock/refs/heads/master/hosts/_malware_typo
>>>>>>>>> 
>>>>>>>>> We strip everything after a # away because we consider it a comment. However, that causes that there is only a line with the domain left which will cause it being listed.
>>>>>>>>> 
>>>>>>>>> The # sign is used as some special character but at the same time it is being used for comments.
>>>>>>>>> 
>>>>>>>>> I will fix this and then refresh the list.
>>>>>>>>> 
>>>>>>>>> -Michael
>>>>>>>>> 
>>>>>>>>>> On 5 Jan 2026, at 11:31, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi Michael,
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On 05/01/2026 12:11, Adolf Belka wrote:
>>>>>>>>>>> Hi Michael,
>>>>>>>>>>> 
>>>>>>>>>>> I have found that the malware list includes duckduckgo.com
>>>>>>>>>>> 
>>>>>>>>>> I have checked through the various sources used for the malware list.
>>>>>>>>>> 
>>>>>>>>>> The ShadowWhisperer (Tracking) list has improving.duckduckgo.com in its list. I suspect that this one is the one causing the problem.
>>>>>>>>>> 
>>>>>>>>>> The mtxadmin (_malware_typo) list has duckduckgo.com mentioned 3 times but not directly as a domain name - looks more like a reference.
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> 
>>>>>>>>>> Adolf.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> Regards,
>>>>>>>>>>> Adolf.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On 02/01/2026 14:02, Adolf Belka wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> 
>>>>>>>>>>>> On 02/01/2026 12:09, Michael Tremer wrote:
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 30 Dec 2025, at 14:05, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 29/12/2025 13:05, Michael Tremer wrote:
>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I hope everyone had a great Christmas and a couple of quiet days to relax from all the stress that was the year 2025.
>>>>>>>>>>>>>> Still relaxing.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Very good, so let’s have a strong start into 2026 now!
>>>>>>>>>>>> 
>>>>>>>>>>>> Starting next week, yes.
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Having a couple of quieter days, I have been working on a new, little (hopefully) side project that has probably been high up on our radar since the Shalla list has shut down in 2020, or maybe even earlier. The goal of the project is to provide good lists with categories of domain names which are usually used to block access to these domains.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> How did we get here?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> As stated before, the URL filter feature in IPFire has the problem that there are not many good blocklists available any more. There used to be a couple more - most famously the Shalla list - but we are now down to a single list from the University of Toulouse. It is a great list, but it is not always the best fit for all users.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Then there has been talk about whether we could implement more blocking features into IPFire that don’t involve the proxy. Most famously blocking over DNS. The problem here remains a the blocking feature is only as good as the data that is fed into it. Some people have been putting forward a number of lists that were suitable for them, but they would not have replaced the blocking functionality as we know it. Their aim is to provide “one list for everything” but that is not what people usually want. It is targeted at a classic home user and the only separation that is being made is any adult/porn/NSFW content which usually is put into a separate list.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> It would have been technically possible to include these lists and let the users decide, but that is not the aim of IPFire. We want to do the job for the user so that their job is getting easier. Including obscure lists that don’t have a clear outline of what they actually want to block (“bad content” is not a category) and passing the burden of figuring out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with cream on top is really not going to work. It is all confusing and will lead to a bad user experience.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> An even bigger problem that is however completely impossible to solve is bad licensing of these lists. A user has asked the publisher of the HaGeZi list whether they could be included in IPFire and under what terms. The response was that the list is available under the terms of the GNU General Public License v3, but that does not seem to be true. The list contains data from various sources. Many of them are licensed under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public agreement that this data may be redistributed, there is a huge legal issue here. We would expose our users to potential copyright infringement which we cannot do under any circumstances. Furthermore many lists are available under a non-commercial license which excludes them from being used in any kind of business. Plenty of IPFire systems are running in businesses, if not even the vast majority.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> In short, these lists are completely unusable for us. Apart from HaGeZi, I consider OISD to have the same problem.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Enough about all the things that are bad. Let’s talk about the new, good things:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Many blacklists on the internet are an amalgamation of other lists. These lists vary in quality with some of them being not that good and without a clear focus and others being excellent data. Since we don’t have the man power to start from scratch, I felt that we can copy the concept that HaGeZi and OISD have started and simply create a new list that is based on other lists at the beginning to have a good starting point. That way, we have much better control over what is going on these lists and we can shape and mould them as we need them. Most importantly, we don’t create a single lists, but many lists that have a clear focus and allow users to choose what they want to block and what not.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> So the current experimental stage that I am in has these lists:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> * Ads
>>>>>>>>>>>>>>> * Dating
>>>>>>>>>>>>>>> * DoH
>>>>>>>>>>>>>>> * Gambling
>>>>>>>>>>>>>>> * Malware
>>>>>>>>>>>>>>> * Porn
>>>>>>>>>>>>>>> * Social
>>>>>>>>>>>>>>> * Violence
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The categories have been determined by what source lists we have available with good data and are compatible with our chosen license CC BY-SA 4.0. This is the same license that we are using for the IPFire Location database, too.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The main use-cases for any kind of blocking are to comply with legal requirements in networks with children (i.e. schools) to remove any kind of pornographic content, sometimes block social media as well. Gambling and violence are commonly blocked, too. Even more common would be filtering advertising and any malicious content.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The latter is especially difficult because so many source lists throw phishing, spyware, malvertising, tracking and other things into the same bucket. Here this is currently all in the malware list which has therefore become quite large. I am not sure whether this will stay like this in the future or if we will have to make some adjustments, but that is exactly why this is now entering some larger testing.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> What has been built so far? In order to put these lists together properly, track any data about where it is coming from, I have built a tool in Python available here:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> https://git.ipfire.org/?p=dnsbl.git;a=summary
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> This tool will automatically update all lists once an hour if there have been any changes and export them in various formats. The exported lists are available for download here:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> https://dnsbl.ipfire.org/lists/
>>>>>>>>>>>>>> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the custom url works fine.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> However you need to remember not to put the https:// at the front of the url otherwise the WUI page completes without any error messages but leaves an error message in the system logs saying
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> URL filter blacklist - ERROR: Not a valid URL filter blacklist
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I found this out the hard way.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Oh yes, I forgot that there is a field on the web UI. If that does not accept https:// as a prefix, please file a bug and we will fix it.
>>>>>>>>>>>> 
>>>>>>>>>>>> I will confirm it and raise a bug.
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The other thing I noticed is that if you already have the Toulouse University list downloaded and you then change to the ipfire custom url then all the existing Toulouse blocklists stay in the directory on IPFire and so you end up with a huge number of category tick boxes, most of which are the old Toulouse ones, which are still available to select and it is not clear which ones are from Toulouse and which ones from IPFire.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Yes, I got the same thing, too. I think this is a bug, too, because otherwise you would have a lot of unused categories lying around that will never be updated. You cannot even tell which ones are from the current list and which ones from the old list.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Long-term we could even consider to remove the Univ. Toulouse list entirely and only have our own lists available which would make the problem go away.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I think if the blocklist URL source is changed or a custom url is provided the first step should be to remove the old ones already existing.
>>>>>>>>>>>>>> That might be a problem because users can also create their own blocklists and I believe those go into the same directory.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Good thought. We of course cannot delete the custom lists.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Without clearing out the old blocklists you end up with a huge number of checkboxes for lists but it is not clear what happens if there is a category that has the same name for the Toulouse list and the IPFire list such as gambling. I will have a look at that and see what happens.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Not sure what the best approach to this is.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I believe it is removing all old content.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Manually deleting all contents of the urlfilter/blacklists/ directory and then selecting the IPFire blocklist url for the custom url I end up with only the 8 categories from the IPFire list.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I have tested some gambling sites from the IPFire list and the block worked on some. On others the site no longer exists so there is nothing to block or has been changed to an https site and in that case it went straight through. Also if I chose the http version of the link, it was automatically changed to https and went through without being blocked.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The entire IPFire infrastructure always requires HTTPS. If you start using HTTP, you will be automatically redirected. It is 2026 and we don’t need to talk HTTP any more :)
>>>>>>>>>>>> 
>>>>>>>>>>>> Some of the domains in the gambling list (maybe quite a lot) seem to only have an http access. If I tried https it came back with the fact that it couldn't find it.
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I am glad to hear that the list is actually blocking. It would have been bad if it didn’t. Now we have the big task to check out the “quality” - however that can be determined. I think this is what needs some time…
>>>>>>>>>>>>> 
>>>>>>>>>>>>> In the meantime I have set up a small page on our website:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> https://www.ipfire.org/dnsbl
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I would like to run this as a first-class project inside IPFire like we are doing with IPFire Location. That means that we need to tell people about what we are doing. Hopefully this page is a little start.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Initially it has a couple of high-level bullet points about what we are trying to achieve. I don’t think the text is very good, yet, but it is the best I had in that moment. There is then also a list of the lists that we currently offer. For each list, a detailed page will tell you about the license, how many domains are listed, when the last update has been, the sources and even there is a history page that shows all the changes whenever they have happened.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Finally there is a section that explains “How To Use?” the list which I would love to extend to include AdGuard Plus and things like that as well as Pi-Hole and whatever else could use the list. In a later step we should go ahead and talk to any projects to include our list(s) into their dropdown so that people can enable them nice and easy.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Behind the web page there is an API service that is running on the host that is running the DNSBL. The frontend web app that is running www.ipfire.org <http://www.ipfire.org/> is connecting to that API service to fetch the current lists, any details and so on. That way, we can split the logic and avoid creating a huge monolith of a web app. This also means that page could be down a little as I am still working on the entire thing and will frequently restart it.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The API documentation is available here and the API is publicly available: https://api.dnsbl.ipfire.org/docs
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The website/API allows to file reports for anything that does not seem to be right on any of the lists. I would like to keep it as an open process, however, long-term, this cannot cost us any time. In the current stage, the reports are getting filed and that is about it. I still need to build out some way for admins or moderators (I am not sure what kind of roles I want to have here) to accept or reject those reports.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> In case of us receiving a domain from a source list, I would rather like to submit a report to upstream for them to de-list. That way, we don’t have any admin to do and we are contributing back to other list. That would be a very good thing to do. We cannot however throw tons of emails at some random upstream projects without co-ordinating this first. By not reporting upstream, we will probably over time create large whitelists and I am not sure if that is a good thing to do.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Finally, there is a search box that can be used to find out if a domain is listed on any of the lists.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> If you download and open any of the files, you will see a large header that includes copyright information and lists all sources that have been used to create the individual lists. This way we ensure maximum transparency, comply with the terms of the individual licenses of the source lists and give credit to the people who help us to put together the most perfect list for our users.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I would like this to become a project that is not only being used in IPFire. We can and will be compatible with other solutions like AdGuard, PiHole so that people can use our lists if they would like to even though they are not using IPFire. Hopefully, these users will also feed back to us so that we can improve our lists over time and make them one of the best options out there.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> All lists are available as a simple text file that lists the domains. Then there is a hosts file available as well as a DNS zone file and an RPZ file. Each list is individually available to be used in squidGuard and there is a larger tarball available with all lists that can be used in IPFire’s URL Filter. I am planning to add Suricata/Snort signatures whenever I have time to do so. Even though it is not a good idea to filter pornographic content this way, I suppose that catching malware and blocking DoH are good use-cases for an IPS. Time will tell…
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> As a start, we will make these lists available in IPFire’s URL Filter and collect some feedback about how we are doing. Afterwards, we can see where else we can take this project.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> If you want to enable this on your system, simply add the URL to your autoupdate.urls file like here:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f
>>>>>>>>>>>>>> I also tested out adding the IPFire url to autoupdate.urls and that also worked fine for me.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Very good. Should we include this already with Core Update 200? I don’t think we would break anything, but we might already gain a couple more people who are helping us to test this all?
>>>>>>>>>>>> 
>>>>>>>>>>>> I think that would be a good idea.
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The next step would be to build and test our DNS infrastructure. In the “How To Use?” Section on the pages of the individual lists, you can already see some instructions on how to use the lists as an RPZ. In comparison to other “providers”, I would prefer if people would be using DNS to fetch the lists. This is simply to push out updates in a cheap way for us and also do it very regularly.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Initially, clients will pull the entire list using AXFR. There is no way around this as they need to have the data in the first place. After that, clients will only need the changes. As you can see in the history, the lists don’t actually change that often. Sometimes only once a day and therefore downloading the entire list again would be a huge waste of data, both on the client side, but also for us hosting then.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Some other providers update their lists “every 10 minutes”, and there won't be any changes whatsoever. We don’t do that. We will only export the lists again when they have actually changed. The timestamps on the files that we offer using HTTPS can be checked by clients so that they won’t re-download the list again if it has not been changed. But using HTTPS still means that we would have to re-download the entire list and not only the changes.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Using DNS and IXFR will update the lists by only transferring a few kilobytes and therefore we can have clients check once an hour if a list has actually changed and only send out the raw changes. That way, we will be able to serve millions of clients at very cheap cost and they will always have a very up to date list.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> As far as I can see any DNS software that supports RPZs supports AXFR/IXFR with exception of Knot Resolver which expects the zone to be downloaded externally. There is a ticket for AXFR/IXFR support (https://gitlab.nic.cz/knot/knot-resolver/-/issues/195).
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Initially, some of the lists have been *huge* which is why a simple HTTP download is not feasible. The porn list was over 100 MiB. We could have spent thousands on just traffic alone which I don’t have for this kind of project. It would also be unnecessary money being spent. There are simply better solutions out there. But then I built something that basically tests the data that we are receiving from upstream but simply checking if a listed domain still exists. The result was very astonishing to me.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> So whenever someone adds a domain to the list, we will (eventually, but not immediately) check if we can resolve the domain’s SOA record. If not, we mark the domain as non-active and will no longer include them in the exported data. This brought down the porn list from just under 5 million domains to just 421k. On the sources page (https://www.ipfire.org/dnsbl/lists/porn/sources) I am listing the percentage of dead domains from each of them and the UT1 list has 94% dead domains. Wow.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> If we cannot resolve the domain, neither can our users. So we would otherwise fill the lists with tons of domains that simply could never be reached. And if they cannot be reached, why would we block them? We would waste bandwidth and a lot of memory on each single client.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The other sources have similarly high rations of dead domains. Most of them are in the 50-80% range. Therefore I am happy that we are doing some extra work here to give our users much better data for their filtering.
>>>>>>>>>>>> 
>>>>>>>>>>>> Removing all dead entries sounds like an excellent step.
>>>>>>>>>>>> 
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> 
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> So, if you like, please go and check out the RPZ blocking with Unbound. Instructions are on the page. I would be happy to hear how this is turning out.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Please let me know if there are any more questions, and I would be glad to answer them.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Happy New Year,
>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>> This email is just a brain dump from me to this list. I would be happy to answer any questions about implementation details, etc. if people are interested. Right now, this email is long enough already…
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> All the best,
>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> -- 
>>>>>>>>>> Sent from my laptop
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>> <suricata-log.jpg><ipfire_dnsbl-violence.rules>
> 

Re: Let's launch our own blocklists...

[Michael Tremer]

Thank you for the feedback. Glad to hear that the changes are working well.

> On 28 Jan 2026, at 16:33, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
> 
> On 26.01.2026 18:18, Michael Tremer wrote:
>> Hello Matthias,
> 
> Hi,
> 
>> Are you running this on Core Update 200?
> 
> Now running on Core 199 - after applying...
> 
>> There were some changes required so that we will extract the datasets from the rules tarballs. I am using a special feature in Suricata so that the lists won’t be using too much memory and will be quickly searchable:
>> 
>>  https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=f0b43241a501f7c545e3cb15f6989e945c60b3e2
> 
> ...these patches.
> 
> No more errors. ;-)
> 
> Best
> Matthias
> 
>> -Michael
>> 
>>> On 25 Jan 2026, at 17:50, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>>> 
>>> On 25.01.2026 15:40, Michael Tremer wrote:
>>>> Hello Matthias,
>>> 
>>> Hi Michael,
>>> 
>>>> Nice catch!
>>>> 
>>>> I fixed it here and added the missing “;”:
>>> 
>>> Yep. The missing ";" seems to be fixed, but 'suricata' still doesn't
>>> like our rules. I added 'violence' as an attachment... ;-)
>>> 
>>>> https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=775561e322ceed43e255e5547bd76047b9f8a40b
>>>> 
>>>> If you go to the provider settings there is a button to force a ruleset update which should give you the fixed version. Please let me know if this works.
>>> 
>>> I did that - rules were updated, but no luck:
>>> 
>>> ***SNIP***
>>> ...
>>> 18:37:37  suricata:  [2577] <Info> -- Including configuration file
>>> /var/ipfire/suricata/suricata-used-rulesfiles.yaml.
>>> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
>>> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop dns
>>> any any -> any any (msg:"IPFire DBL [Violence] Blocked DNS Query";
>>> dns.query; domain; dataset:isset,violence,type string,load
>>> datasets/violence.txt; classtype:policy-violation; priority:2;
>>> sid:1048577; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
>>> metadata:dbl violence.dbl.ipfire.org;)" from file
>>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 39
>>> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
>>> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop
>>> http any any -> any any (msg:"IPFire DBL [Violence] Blocked HTTP
>>> Request"; http.host; domain; dataset:isset,violence,type string,load
>>> datasets/violence.txt; classtype:policy-violation; priority:2;
>>> sid:1048578; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
>>> metadata:dbl violence.dbl.ipfire.org;)" from file
>>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 40
>>> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
>>> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop tls
>>> any any -> any any (msg:"IPFire DBL [Violence] Blocked TLS Connection";
>>> tls.sni; domain; dataset:isset,violence,type string,load
>>> datasets/violence.txt; classtype:policy-violation; priority:2;
>>> sid:1048579; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
>>> metadata:dbl violence.dbl.ipfire.org;)" from file
>>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 41
>>> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
>>> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop
>>> quic any any -> any any (msg:"IPFire DBL [Violence] Blocked QUIC
>>> Connection"; quic.sni; domain; dataset:isset,violence,type string,load
>>> datasets/violence.txt; classtype:policy-violation; priority:2;
>>> sid:1048580; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
>>> metadata:dbl violence.dbl.ipfire.org;)" from file
>>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 42
>>> ...
>>> ***SNAP***
>>> 
>>> For better reading - see attached screenshot.
>>> 
>>> Best
>>> Matthias
>>> 
>>>> Best,
>>>> -Michael
>>>> 
>>>>> On 24 Jan 2026, at 23:41, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>>>>> 
>>>>> On 23.01.2026 17:39, Michael Tremer wrote:
>>>>>> Hello Matthias,
>>>>> 
>>>>> Hi Michael,
>>>>> 
>>>>>> Thank you very much for testing IPFire DBL.
>>>>> 
>>>>> No problem - I have news:
>>>>> 
>>>>> After taking a closer look to the IPS system logs, unfortunately I found
>>>>> some parsing errors:
>>>>> 
>>>>> 'suricata' complains about missing ";".
>>>>> 
>>>>> ***SNIP***
>>>>> ...
>>>>> 00:32:40 suricata: [13343] <Info> -- Including configuration file
>>>>> /var/ipfire/suricata/suricata-used-rulesfiles.yaml.
>>>>> 00:32:40 suricata: [13343] <Error> -- no terminating ";" found
>>>>> 00:32:40 suricata: [13343] <Error> -- error parsing signature "drop
>>>>> dns any any -> any any (msg:"IPFire DBL [Advertising] Blocked DNS
>>>>> Query"; dns.query; domain; dataset:isset,ads,type string,load
>>>>> datasets/ads.txt; classtype:policy-violation; priority:3; sid:983041;
>>>>> rev:1; reference:url,https://www.ipfire.org/dbl/ads; metadata:dbl
>>>>> ads.dbl.ipfire.org)" from file /var/lib/suricata/ipfire_dnsbl-ads.rules
>>>>> at line 72
>>>>> 00:32:40 suricata: [13343] <Error> -- no terminating ";" found
>>>>> ...
>>>>> ***SNAP***
>>>>> 
>>>>> I tried, but didn't find the right place for any missing ";".
>>>>> 
>>>>> Can "anyone" confirm?
>>>>> 
>>>>> Best
>>>>> Matthias
>>>>> 
>>>>>>> On 23 Jan 2026, at 15:02, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>>>>>>> 
>>>>>>> On 22.01.2026 12:33, Michael Tremer wrote:
>>>>>>>> Hello everyone,
>>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> short feedback from me:
>>>>>>> 
>>>>>>> - I activated both the suricata (IPFire DBL - Domain Blocklist) - and
>>>>>>> the URLfilter lists from 'dbl.ipfire.org'.
>>>>>> 
>>>>>> This is an interesting case. What I didn’t manage to test yet is what happens when Suricata blocks the connection first. If URL Filter sees a domain that is being blocked it will either send you an error page if you are using HTTP, or simply close the connection if it is HTTPS. However, when Suricata comes first in the chain (and it will), it might close the connection because URL Filter has received the request. In the case of HTTPS this does not make any difference because the connection will be closed, but in the HTTP case you won’t see an error page any more and instead have the connection closed, too. You are basically losing the explicit error notification which is a little bit annoying.
>>>>>> 
>>>>>> We could have the same when we are doing the same with Unbound and DNS filtering. Potentially we would need to whitelist the local DNS resolver then, but how is Suricata supposed to know that the same categories are activated in both places?
>>>>>> 
>>>>>>> - I even took the 'smart-tv' domains from the IFire DBL blacklist and
>>>>>>> copied/pasted them in my fritzbox filter lists.
>>>>>> 
>>>>>> LOL Why not use IPFire to filter this as well?
>>>>>> 
>>>>>>> Everything works as expected. Besides, the download of the IPFire
>>>>>>> DBL-list loads a lot faster than the list from 'Univ. Toulouse'... ;-)
>>>>>> 
>>>>>> Yes, we don’t have much traffic on the server, yet.
>>>>>> 
>>>>>>> Functionality is good - no false positives or seen problems. Good work -
>>>>>>> thanks!
>>>>>> 
>>>>>> Nice. We need to distinguish a little between what is a technical issue and what is a false-positive/missing domain on the list. However, testing both at the same time is something we will all cope quite well with :)
>>>>>> 
>>>>>> -Michael
>>>>>> 
>>>>>>> Best
>>>>>>> Matthias
>>>>>>> 
>>>>>>>> Over the past few weeks I have made significant progress on this all, and I think we're getting close to something the community will be really happy with. I'd love to get feedback from the team before we finalise things.
>>>>>>>> 
>>>>>>>> So what has happened?
>>>>>>>> 
>>>>>>>> First of all, the entire project has been renamed. DNSBL is not entirely what this is. Although the lists can be thrown into DNS, they have much more use outside of it that I thought we should simply go with DBL, short for Domain Blocklist. After all, we are only importing domains. The new home of the project therefore is https://www.ipfire.org/dbl
>>>>>>>> 
>>>>>>>> I have added a couple more lists that I thought interesting and I have added a couple more sources that I considered a good start. Hopefully, we will soon gather some more feedback on how well this is all holding up. My main focus has however been on the technology that will power this project.
>>>>>>>> 
>>>>>>>> One of the bigger challenges was to create Suricata rules from the lists. Initially I tried to create a ton of rules but since our lists are so large, this quickly became too complicated. I have now settled on using a feature that is only available in more recent versions of Suricata (I believe 7 and later), but since we are already on Suricata 8 in IPFire this won’t be a problem for us. All domains for each list are basically compiled into one massively large dataset and one single rule is referring to that dataset. This way, we won’t have the option to remove any false-positives, but at least Suricata and the GUI won’t starve a really bad death when loading millions of rules.
>>>>>>>> 
>>>>>>>> Suricata will now be able to use our rules to block access to any listed domains of each of the categories over DNS, HTTP, TLS or QUIC. Although I don’t expect many users to use Suricata to block porn or other things, this is a great backstop to enforce any policy like that. For example, if there is a user on the network who is trying to circumvent the DNS server that might filter out certain domains, even after getting an IP address resolved through other means, they won’t be able to open a TLS/QUIC connection or send a HTTP request to all blocked domains. Some people have said they were interested in blocking DNS-over-HTTPS and this is a perfect way to do this and actually be sure that any server that is being blocked on the list will actually be completely inaccessible.
>>>>>>>> 
>>>>>>>> Those Suricata rules are already available for testing in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=9eb8751487d23dd354a105c28bdbbb0398fe6e85
>>>>>>>> 
>>>>>>>> I have chosen various severities for the lists. If someone was to block advertising using DBL, this is fine, but not a very severe alert. If someone chooses to block malware and there is a system on the network trying to access those domains, this is an alert worth being investigated by an admin. Our new Suricata Reporter will show those violations in different colours based on the severity which helps to identify the right alerts to further investigate.
>>>>>>>> 
>>>>>>>> Formerly I have asked you to test the lists using URL Filter. Those rules are now available as well in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=db160694279a4b10378447f775dd536fdfcfb02a
>>>>>>>> 
>>>>>>>> I talked about a method to remove any dead domains from any sources which is a great way to keep our lists smaller. The pure size of them is a problem in so many ways. That check was however a little bit too ambitious and I had to make it a little bit less eager. Basically if we are in doubt, we need to still list the domain because it might be resolvable by a user.
>>>>>>>> 
>>>>>>>> https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=bb5b6e33b731501d45dea293505f7d42a61d5ce7
>>>>>>>> 
>>>>>>>> So how else could we make the lists smaller without losing any actual data? Since we sometimes list a whole TLD (e.g. .xxx or .porn), there is very little point in listing any domains of this TLD. They will always be caught anyways. So I built a check that marks all domains that don’t need to be included on the exported lists because they will never be needed and was able to shrink the size of the lists by a lot again.
>>>>>>>> 
>>>>>>>> The website does not show this data, but the API returns the number of “subsumed” domains (I didn’t have a better name):
>>>>>>>> 
>>>>>>>> curl https://api.dbl.ipfire.org/lists | jq .
>>>>>>>> 
>>>>>>>> The number shown would normally be added to the total number of domains and usually cuts the size of the list by 50-200%.
>>>>>>>> 
>>>>>>>> Those stats will now also be stored in a history table so that we will be able to track growth of all lists.
>>>>>>>> 
>>>>>>>> Furthermore, the application will now send email notifications for any incoming reports. This way, we will be able to stay in close touch with the reporters and keep them up to date on their submissions as well as inform moderators that there is something to have a look at.
>>>>>>>> 
>>>>>>>> The search has been refactored as well, so that we can show clearly whether something is blocked or not at one glance: https://www.ipfire.org/dbl/search?q=github.com. There is detailed information available on all domains and what happened to them. In case of GitHub.com, this seems to be blocked and unblocked by someone all of the time and we can see a clear audit trail of that: https://www.ipfire.org/dbl/lists/malware/domains/github.com
>>>>>>>> 
>>>>>>>> On the DNS front, I have added some metadata to the zones so that people can programmatically request some data, like when it has been last updated (in a human-friendly timestamp and not only the serial), license, description and so on:
>>>>>>>> 
>>>>>>>> # dig +short ANY _info.ads.dbl.ipfire.org @primary.dbl.ipfire.org
>>>>>>>> "total-domains=42226"
>>>>>>>> "license=CC BY-SA 4.0"
>>>>>>>> "updated-at=2026-01-20T22:17:02.409933+00:00"
>>>>>>>> "description=Blocks domains used for ads, tracking, and ad delivery”
>>>>>>>> 
>>>>>>>> Now, I would like to hear more feedback from you. I know we've all been stretched thin lately, so I especially appreciate anyone who has time to review and provide input. Ideas, just say if you like it or not. Where this could go in the future?
>>>>>>>> 
>>>>>>>> Looking ahead, I would like us to start thinking about the RPZ feature that has been on the wishlist. IPFire DBL has been a bigger piece of work, and I think it's worth having a conversation about sustainability. Resources for this need to be allocated and paid for. Open source is about freedom, not free beer — and to keep building features like this, we will need to explore some funding options. I would be interested to hear any ideas you might have that could work for IPFire.
>>>>>>>> 
>>>>>>>> Please share your thoughts on the mailing list when you can — even a quick 'looks good' or 'I have concerns about X' is valuable. Public discussion helps everyone stay in the loop and contribute.
>>>>>>>> 
>>>>>>>> I am aiming to move forward with this in a week's time, so if you have input, now would be a good time to share it.
>>>>>>>> 
>>>>>>>> Best,
>>>>>>>> -Michael
>>>>>>>> 
>>>>>>>>> On 6 Jan 2026, at 10:20, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>>>>>>>> 
>>>>>>>>> Good Morning Adolf,
>>>>>>>>> 
>>>>>>>>> I had a look at this problem yesterday and it seems that parsing the format is becoming a little bit difficult this way. Since this is only affecting very few domains, I have simply whitelisted them all manually and duckduckgo.com <http://duckduckgo.com/> and others should now be easily reachable again.
>>>>>>>>> 
>>>>>>>>> Please let me know if you have any more findings.
>>>>>>>>> 
>>>>>>>>> All the best,
>>>>>>>>> -Michael
>>>>>>>>> 
>>>>>>>>>> On 5 Jan 2026, at 11:48, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hello Adolf,
>>>>>>>>>> 
>>>>>>>>>> This is a good find.
>>>>>>>>>> 
>>>>>>>>>> But if duckduckgo.com <http://duckduckgo.com/> is blocked, we will have to have a source somewhere that blocks that domain. Not only a sub-domain of it. Otherwise we have a bug somewhere.
>>>>>>>>>> 
>>>>>>>>>> This is most likely as the domain is listed here, but with some stuff afterwards:
>>>>>>>>>> 
>>>>>>>>>> https://raw.githubusercontent.com/mtxadmin/ublock/refs/heads/master/hosts/_malware_typo
>>>>>>>>>> 
>>>>>>>>>> We strip everything after a # away because we consider it a comment. However, that causes that there is only a line with the domain left which will cause it being listed.
>>>>>>>>>> 
>>>>>>>>>> The # sign is used as some special character but at the same time it is being used for comments.
>>>>>>>>>> 
>>>>>>>>>> I will fix this and then refresh the list.
>>>>>>>>>> 
>>>>>>>>>> -Michael
>>>>>>>>>> 
>>>>>>>>>>> On 5 Jan 2026, at 11:31, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Hi Michael,
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On 05/01/2026 12:11, Adolf Belka wrote:
>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>> 
>>>>>>>>>>>> I have found that the malware list includes duckduckgo.com
>>>>>>>>>>>> 
>>>>>>>>>>> I have checked through the various sources used for the malware list.
>>>>>>>>>>> 
>>>>>>>>>>> The ShadowWhisperer (Tracking) list has improving.duckduckgo.com in its list. I suspect that this one is the one causing the problem.
>>>>>>>>>>> 
>>>>>>>>>>> The mtxadmin (_malware_typo) list has duckduckgo.com mentioned 3 times but not directly as a domain name - looks more like a reference.
>>>>>>>>>>> 
>>>>>>>>>>> Regards,
>>>>>>>>>>> 
>>>>>>>>>>> Adolf.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On 02/01/2026 14:02, Adolf Belka wrote:
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 02/01/2026 12:09, Michael Tremer wrote:
>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 30 Dec 2025, at 14:05, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 29/12/2025 13:05, Michael Tremer wrote:
>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I hope everyone had a great Christmas and a couple of quiet days to relax from all the stress that was the year 2025.
>>>>>>>>>>>>>>> Still relaxing.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Very good, so let’s have a strong start into 2026 now!
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Starting next week, yes.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Having a couple of quieter days, I have been working on a new, little (hopefully) side project that has probably been high up on our radar since the Shalla list has shut down in 2020, or maybe even earlier. The goal of the project is to provide good lists with categories of domain names which are usually used to block access to these domains.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> How did we get here?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> As stated before, the URL filter feature in IPFire has the problem that there are not many good blocklists available any more. There used to be a couple more - most famously the Shalla list - but we are now down to a single list from the University of Toulouse. It is a great list, but it is not always the best fit for all users.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Then there has been talk about whether we could implement more blocking features into IPFire that don’t involve the proxy. Most famously blocking over DNS. The problem here remains a the blocking feature is only as good as the data that is fed into it. Some people have been putting forward a number of lists that were suitable for them, but they would not have replaced the blocking functionality as we know it. Their aim is to provide “one list for everything” but that is not what people usually want. It is targeted at a classic home user and the only separation that is being made is any adult/porn/NSFW content which usually is put into a separate list.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> It would have been technically possible to include these lists and let the users decide, but that is not the aim of IPFire. We want to do the job for the user so that their job is getting easier. Including obscure lists that don’t have a clear outline of what they actually want to block (“bad content” is not a category) and passing the burden of figuring out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with cream on top is really not going to work. It is all confusing and will lead to a bad user experience.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> An even bigger problem that is however completely impossible to solve is bad licensing of these lists. A user has asked the publisher of the HaGeZi list whether they could be included in IPFire and under what terms. The response was that the list is available under the terms of the GNU General Public License v3, but that does not seem to be true. The list contains data from various sources. Many of them are licensed under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public agreement that this data may be redistributed, there is a huge legal issue here. We would expose our users to potential copyright infringement which we cannot do under any circumstances. Furthermore many lists are available under a non-commercial license which excludes them from being used in any kind of business. Plenty of IPFire systems are running in businesses, if not even the vast majority.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> In short, these lists are completely unusable for us. Apart from HaGeZi, I consider OISD to have the same problem.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Enough about all the things that are bad. Let’s talk about the new, good things:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Many blacklists on the internet are an amalgamation of other lists. These lists vary in quality with some of them being not that good and without a clear focus and others being excellent data. Since we don’t have the man power to start from scratch, I felt that we can copy the concept that HaGeZi and OISD have started and simply create a new list that is based on other lists at the beginning to have a good starting point. That way, we have much better control over what is going on these lists and we can shape and mould them as we need them. Most importantly, we don’t create a single lists, but many lists that have a clear focus and allow users to choose what they want to block and what not.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> So the current experimental stage that I am in has these lists:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> * Ads
>>>>>>>>>>>>>>>> * Dating
>>>>>>>>>>>>>>>> * DoH
>>>>>>>>>>>>>>>> * Gambling
>>>>>>>>>>>>>>>> * Malware
>>>>>>>>>>>>>>>> * Porn
>>>>>>>>>>>>>>>> * Social
>>>>>>>>>>>>>>>> * Violence
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The categories have been determined by what source lists we have available with good data and are compatible with our chosen license CC BY-SA 4.0. This is the same license that we are using for the IPFire Location database, too.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The main use-cases for any kind of blocking are to comply with legal requirements in networks with children (i.e. schools) to remove any kind of pornographic content, sometimes block social media as well. Gambling and violence are commonly blocked, too. Even more common would be filtering advertising and any malicious content.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The latter is especially difficult because so many source lists throw phishing, spyware, malvertising, tracking and other things into the same bucket. Here this is currently all in the malware list which has therefore become quite large. I am not sure whether this will stay like this in the future or if we will have to make some adjustments, but that is exactly why this is now entering some larger testing.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> What has been built so far? In order to put these lists together properly, track any data about where it is coming from, I have built a tool in Python available here:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> https://git.ipfire.org/?p=dnsbl.git;a=summary
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> This tool will automatically update all lists once an hour if there have been any changes and export them in various formats. The exported lists are available for download here:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> https://dnsbl.ipfire.org/lists/
>>>>>>>>>>>>>>> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the custom url works fine.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> However you need to remember not to put the https:// at the front of the url otherwise the WUI page completes without any error messages but leaves an error message in the system logs saying
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> URL filter blacklist - ERROR: Not a valid URL filter blacklist
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I found this out the hard way.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Oh yes, I forgot that there is a field on the web UI. If that does not accept https:// as a prefix, please file a bug and we will fix it.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I will confirm it and raise a bug.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The other thing I noticed is that if you already have the Toulouse University list downloaded and you then change to the ipfire custom url then all the existing Toulouse blocklists stay in the directory on IPFire and so you end up with a huge number of category tick boxes, most of which are the old Toulouse ones, which are still available to select and it is not clear which ones are from Toulouse and which ones from IPFire.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Yes, I got the same thing, too. I think this is a bug, too, because otherwise you would have a lot of unused categories lying around that will never be updated. You cannot even tell which ones are from the current list and which ones from the old list.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Long-term we could even consider to remove the Univ. Toulouse list entirely and only have our own lists available which would make the problem go away.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I think if the blocklist URL source is changed or a custom url is provided the first step should be to remove the old ones already existing.
>>>>>>>>>>>>>>> That might be a problem because users can also create their own blocklists and I believe those go into the same directory.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Good thought. We of course cannot delete the custom lists.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Without clearing out the old blocklists you end up with a huge number of checkboxes for lists but it is not clear what happens if there is a category that has the same name for the Toulouse list and the IPFire list such as gambling. I will have a look at that and see what happens.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Not sure what the best approach to this is.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I believe it is removing all old content.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Manually deleting all contents of the urlfilter/blacklists/ directory and then selecting the IPFire blocklist url for the custom url I end up with only the 8 categories from the IPFire list.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I have tested some gambling sites from the IPFire list and the block worked on some. On others the site no longer exists so there is nothing to block or has been changed to an https site and in that case it went straight through. Also if I chose the http version of the link, it was automatically changed to https and went through without being blocked.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The entire IPFire infrastructure always requires HTTPS. If you start using HTTP, you will be automatically redirected. It is 2026 and we don’t need to talk HTTP any more :)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Some of the domains in the gambling list (maybe quite a lot) seem to only have an http access. If I tried https it came back with the fact that it couldn't find it.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I am glad to hear that the list is actually blocking. It would have been bad if it didn’t. Now we have the big task to check out the “quality” - however that can be determined. I think this is what needs some time…
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> In the meantime I have set up a small page on our website:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> https://www.ipfire.org/dnsbl
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I would like to run this as a first-class project inside IPFire like we are doing with IPFire Location. That means that we need to tell people about what we are doing. Hopefully this page is a little start.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Initially it has a couple of high-level bullet points about what we are trying to achieve. I don’t think the text is very good, yet, but it is the best I had in that moment. There is then also a list of the lists that we currently offer. For each list, a detailed page will tell you about the license, how many domains are listed, when the last update has been, the sources and even there is a history page that shows all the changes whenever they have happened.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Finally there is a section that explains “How To Use?” the list which I would love to extend to include AdGuard Plus and things like that as well as Pi-Hole and whatever else could use the list. In a later step we should go ahead and talk to any projects to include our list(s) into their dropdown so that people can enable them nice and easy.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Behind the web page there is an API service that is running on the host that is running the DNSBL. The frontend web app that is running www.ipfire.org <http://www.ipfire.org/> is connecting to that API service to fetch the current lists, any details and so on. That way, we can split the logic and avoid creating a huge monolith of a web app. This also means that page could be down a little as I am still working on the entire thing and will frequently restart it.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The API documentation is available here and the API is publicly available: https://api.dnsbl.ipfire.org/docs
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The website/API allows to file reports for anything that does not seem to be right on any of the lists. I would like to keep it as an open process, however, long-term, this cannot cost us any time. In the current stage, the reports are getting filed and that is about it. I still need to build out some way for admins or moderators (I am not sure what kind of roles I want to have here) to accept or reject those reports.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> In case of us receiving a domain from a source list, I would rather like to submit a report to upstream for them to de-list. That way, we don’t have any admin to do and we are contributing back to other list. That would be a very good thing to do. We cannot however throw tons of emails at some random upstream projects without co-ordinating this first. By not reporting upstream, we will probably over time create large whitelists and I am not sure if that is a good thing to do.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Finally, there is a search box that can be used to find out if a domain is listed on any of the lists.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> If you download and open any of the files, you will see a large header that includes copyright information and lists all sources that have been used to create the individual lists. This way we ensure maximum transparency, comply with the terms of the individual licenses of the source lists and give credit to the people who help us to put together the most perfect list for our users.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I would like this to become a project that is not only being used in IPFire. We can and will be compatible with other solutions like AdGuard, PiHole so that people can use our lists if they would like to even though they are not using IPFire. Hopefully, these users will also feed back to us so that we can improve our lists over time and make them one of the best options out there.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> All lists are available as a simple text file that lists the domains. Then there is a hosts file available as well as a DNS zone file and an RPZ file. Each list is individually available to be used in squidGuard and there is a larger tarball available with all lists that can be used in IPFire’s URL Filter. I am planning to add Suricata/Snort signatures whenever I have time to do so. Even though it is not a good idea to filter pornographic content this way, I suppose that catching malware and blocking DoH are good use-cases for an IPS. Time will tell…
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> As a start, we will make these lists available in IPFire’s URL Filter and collect some feedback about how we are doing. Afterwards, we can see where else we can take this project.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> If you want to enable this on your system, simply add the URL to your autoupdate.urls file like here:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f
>>>>>>>>>>>>>>> I also tested out adding the IPFire url to autoupdate.urls and that also worked fine for me.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Very good. Should we include this already with Core Update 200? I don’t think we would break anything, but we might already gain a couple more people who are helping us to test this all?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I think that would be a good idea.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The next step would be to build and test our DNS infrastructure. In the “How To Use?” Section on the pages of the individual lists, you can already see some instructions on how to use the lists as an RPZ. In comparison to other “providers”, I would prefer if people would be using DNS to fetch the lists. This is simply to push out updates in a cheap way for us and also do it very regularly.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Initially, clients will pull the entire list using AXFR. There is no way around this as they need to have the data in the first place. After that, clients will only need the changes. As you can see in the history, the lists don’t actually change that often. Sometimes only once a day and therefore downloading the entire list again would be a huge waste of data, both on the client side, but also for us hosting then.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Some other providers update their lists “every 10 minutes”, and there won't be any changes whatsoever. We don’t do that. We will only export the lists again when they have actually changed. The timestamps on the files that we offer using HTTPS can be checked by clients so that they won’t re-download the list again if it has not been changed. But using HTTPS still means that we would have to re-download the entire list and not only the changes.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Using DNS and IXFR will update the lists by only transferring a few kilobytes and therefore we can have clients check once an hour if a list has actually changed and only send out the raw changes. That way, we will be able to serve millions of clients at very cheap cost and they will always have a very up to date list.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> As far as I can see any DNS software that supports RPZs supports AXFR/IXFR with exception of Knot Resolver which expects the zone to be downloaded externally. There is a ticket for AXFR/IXFR support (https://gitlab.nic.cz/knot/knot-resolver/-/issues/195).
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Initially, some of the lists have been *huge* which is why a simple HTTP download is not feasible. The porn list was over 100 MiB. We could have spent thousands on just traffic alone which I don’t have for this kind of project. It would also be unnecessary money being spent. There are simply better solutions out there. But then I built something that basically tests the data that we are receiving from upstream but simply checking if a listed domain still exists. The result was very astonishing to me.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> So whenever someone adds a domain to the list, we will (eventually, but not immediately) check if we can resolve the domain’s SOA record. If not, we mark the domain as non-active and will no longer include them in the exported data. This brought down the porn list from just under 5 million domains to just 421k. On the sources page (https://www.ipfire.org/dnsbl/lists/porn/sources) I am listing the percentage of dead domains from each of them and the UT1 list has 94% dead domains. Wow.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> If we cannot resolve the domain, neither can our users. So we would otherwise fill the lists with tons of domains that simply could never be reached. And if they cannot be reached, why would we block them? We would waste bandwidth and a lot of memory on each single client.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The other sources have similarly high rations of dead domains. Most of them are in the 50-80% range. Therefore I am happy that we are doing some extra work here to give our users much better data for their filtering.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Removing all dead entries sounds like an excellent step.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> So, if you like, please go and check out the RPZ blocking with Unbound. Instructions are on the page. I would be happy to hear how this is turning out.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Please let me know if there are any more questions, and I would be glad to answer them.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Happy New Year,
>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>> This email is just a brain dump from me to this list. I would be happy to answer any questions about implementation details, etc. if people are interested. Right now, this email is long enough already…
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> All the best,
>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> -- 
>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> <suricata-log.jpg><ipfire_dnsbl-violence.rules>
>> 
> 

Re: Let's launch our own blocklists...

[Matthias Fischer]

Hi,

On 28.01.2026 17:59, Michael Tremer wrote:
> Thank you for the feedback. Glad to hear that the changes are working well.

They are working a bit too good...

I had to deactivate the TLS and HTTP Rules in 'Advertising' and
'Malware' because my proxy got blocked:

***SNIP***
...
Date: 01/28 17:45:39
Name: IPFire DBL [Malware] Blocked HTTP Request
Priority: 1
Type: Potential Corporate Privacy Violation
IP Info: 192.168.XXX.YYY:50132 -> 192.168.XXX.YYY:8080
SID: 786434
Refs:
...
Date: 01/28 17:45:39
Name: IPFire DBL [Advertising] Blocked HTTP Request
Priority: 3
Type: Potential Corporate Privacy Violation
IP Info: 192.168.XXX.YYY:50133 -> 192.168.XXX.YYY:8080
SID: 983042
Refs:
...
Date: 01/28 18:55:01
Name: IPFire DBL [Advertising] Blocked TLS Connection
Priority: 3
Type: Potential Corporate Privacy Violation
IP Info: 192.168.XXX.YYY:51061 -> 192.168.XXX.YYY:8080
SID: 983043
Refs:
...
Date: 01/28 21:15:47
Name: IPFire DBL [Malware] Blocked TLS Connection
Priority: 1
Type: Potential Corporate Privacy Violation
IP Info: 192.168.XXX.YYY:51766 -> 192.168.XXX.YYY:8080
SID: 786435
Refs:
...
***SNAP***

Best
Matthias

>> On 28 Jan 2026, at 16:33, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>> 
>> On 26.01.2026 18:18, Michael Tremer wrote:
>>> Hello Matthias,
>> 
>> Hi,
>> 
>>> Are you running this on Core Update 200?
>> 
>> Now running on Core 199 - after applying...
>> 
>>> There were some changes required so that we will extract the datasets from the rules tarballs. I am using a special feature in Suricata so that the lists won’t be using too much memory and will be quickly searchable:
>>> 
>>>  https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=f0b43241a501f7c545e3cb15f6989e945c60b3e2
>> 
>> ...these patches.
>> 
>> No more errors. ;-)
>> 
>> Best
>> Matthias
>> 
>>> -Michael
>>> 
>>>> On 25 Jan 2026, at 17:50, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>>>> 
>>>> On 25.01.2026 15:40, Michael Tremer wrote:
>>>>> Hello Matthias,
>>>> 
>>>> Hi Michael,
>>>> 
>>>>> Nice catch!
>>>>> 
>>>>> I fixed it here and added the missing “;”:
>>>> 
>>>> Yep. The missing ";" seems to be fixed, but 'suricata' still doesn't
>>>> like our rules. I added 'violence' as an attachment... ;-)
>>>> 
>>>>> https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=775561e322ceed43e255e5547bd76047b9f8a40b
>>>>> 
>>>>> If you go to the provider settings there is a button to force a ruleset update which should give you the fixed version. Please let me know if this works.
>>>> 
>>>> I did that - rules were updated, but no luck:
>>>> 
>>>> ***SNIP***
>>>> ...
>>>> 18:37:37  suricata:  [2577] <Info> -- Including configuration file
>>>> /var/ipfire/suricata/suricata-used-rulesfiles.yaml.
>>>> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
>>>> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop dns
>>>> any any -> any any (msg:"IPFire DBL [Violence] Blocked DNS Query";
>>>> dns.query; domain; dataset:isset,violence,type string,load
>>>> datasets/violence.txt; classtype:policy-violation; priority:2;
>>>> sid:1048577; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
>>>> metadata:dbl violence.dbl.ipfire.org;)" from file
>>>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 39
>>>> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
>>>> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop
>>>> http any any -> any any (msg:"IPFire DBL [Violence] Blocked HTTP
>>>> Request"; http.host; domain; dataset:isset,violence,type string,load
>>>> datasets/violence.txt; classtype:policy-violation; priority:2;
>>>> sid:1048578; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
>>>> metadata:dbl violence.dbl.ipfire.org;)" from file
>>>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 40
>>>> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
>>>> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop tls
>>>> any any -> any any (msg:"IPFire DBL [Violence] Blocked TLS Connection";
>>>> tls.sni; domain; dataset:isset,violence,type string,load
>>>> datasets/violence.txt; classtype:policy-violation; priority:2;
>>>> sid:1048579; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
>>>> metadata:dbl violence.dbl.ipfire.org;)" from file
>>>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 41
>>>> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
>>>> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop
>>>> quic any any -> any any (msg:"IPFire DBL [Violence] Blocked QUIC
>>>> Connection"; quic.sni; domain; dataset:isset,violence,type string,load
>>>> datasets/violence.txt; classtype:policy-violation; priority:2;
>>>> sid:1048580; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
>>>> metadata:dbl violence.dbl.ipfire.org;)" from file
>>>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 42
>>>> ...
>>>> ***SNAP***
>>>> 
>>>> For better reading - see attached screenshot.
>>>> 
>>>> Best
>>>> Matthias
>>>> 
>>>>> Best,
>>>>> -Michael
>>>>> 
>>>>>> On 24 Jan 2026, at 23:41, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>>>>>> 
>>>>>> On 23.01.2026 17:39, Michael Tremer wrote:
>>>>>>> Hello Matthias,
>>>>>> 
>>>>>> Hi Michael,
>>>>>> 
>>>>>>> Thank you very much for testing IPFire DBL.
>>>>>> 
>>>>>> No problem - I have news:
>>>>>> 
>>>>>> After taking a closer look to the IPS system logs, unfortunately I found
>>>>>> some parsing errors:
>>>>>> 
>>>>>> 'suricata' complains about missing ";".
>>>>>> 
>>>>>> ***SNIP***
>>>>>> ...
>>>>>> 00:32:40 suricata: [13343] <Info> -- Including configuration file
>>>>>> /var/ipfire/suricata/suricata-used-rulesfiles.yaml.
>>>>>> 00:32:40 suricata: [13343] <Error> -- no terminating ";" found
>>>>>> 00:32:40 suricata: [13343] <Error> -- error parsing signature "drop
>>>>>> dns any any -> any any (msg:"IPFire DBL [Advertising] Blocked DNS
>>>>>> Query"; dns.query; domain; dataset:isset,ads,type string,load
>>>>>> datasets/ads.txt; classtype:policy-violation; priority:3; sid:983041;
>>>>>> rev:1; reference:url,https://www.ipfire.org/dbl/ads; metadata:dbl
>>>>>> ads.dbl.ipfire.org)" from file /var/lib/suricata/ipfire_dnsbl-ads.rules
>>>>>> at line 72
>>>>>> 00:32:40 suricata: [13343] <Error> -- no terminating ";" found
>>>>>> ...
>>>>>> ***SNAP***
>>>>>> 
>>>>>> I tried, but didn't find the right place for any missing ";".
>>>>>> 
>>>>>> Can "anyone" confirm?
>>>>>> 
>>>>>> Best
>>>>>> Matthias
>>>>>> 
>>>>>>>> On 23 Jan 2026, at 15:02, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>>>>>>>> 
>>>>>>>> On 22.01.2026 12:33, Michael Tremer wrote:
>>>>>>>>> Hello everyone,
>>>>>>>> 
>>>>>>>> Hi,
>>>>>>>> 
>>>>>>>> short feedback from me:
>>>>>>>> 
>>>>>>>> - I activated both the suricata (IPFire DBL - Domain Blocklist) - and
>>>>>>>> the URLfilter lists from 'dbl.ipfire.org'.
>>>>>>> 
>>>>>>> This is an interesting case. What I didn’t manage to test yet is what happens when Suricata blocks the connection first. If URL Filter sees a domain that is being blocked it will either send you an error page if you are using HTTP, or simply close the connection if it is HTTPS. However, when Suricata comes first in the chain (and it will), it might close the connection because URL Filter has received the request. In the case of HTTPS this does not make any difference because the connection will be closed, but in the HTTP case you won’t see an error page any more and instead have the connection closed, too. You are basically losing the explicit error notification which is a little bit annoying.
>>>>>>> 
>>>>>>> We could have the same when we are doing the same with Unbound and DNS filtering. Potentially we would need to whitelist the local DNS resolver then, but how is Suricata supposed to know that the same categories are activated in both places?
>>>>>>> 
>>>>>>>> - I even took the 'smart-tv' domains from the IFire DBL blacklist and
>>>>>>>> copied/pasted them in my fritzbox filter lists.
>>>>>>> 
>>>>>>> LOL Why not use IPFire to filter this as well?
>>>>>>> 
>>>>>>>> Everything works as expected. Besides, the download of the IPFire
>>>>>>>> DBL-list loads a lot faster than the list from 'Univ. Toulouse'... ;-)
>>>>>>> 
>>>>>>> Yes, we don’t have much traffic on the server, yet.
>>>>>>> 
>>>>>>>> Functionality is good - no false positives or seen problems. Good work -
>>>>>>>> thanks!
>>>>>>> 
>>>>>>> Nice. We need to distinguish a little between what is a technical issue and what is a false-positive/missing domain on the list. However, testing both at the same time is something we will all cope quite well with :)
>>>>>>> 
>>>>>>> -Michael
>>>>>>> 
>>>>>>>> Best
>>>>>>>> Matthias
>>>>>>>> 
>>>>>>>>> Over the past few weeks I have made significant progress on this all, and I think we're getting close to something the community will be really happy with. I'd love to get feedback from the team before we finalise things.
>>>>>>>>> 
>>>>>>>>> So what has happened?
>>>>>>>>> 
>>>>>>>>> First of all, the entire project has been renamed. DNSBL is not entirely what this is. Although the lists can be thrown into DNS, they have much more use outside of it that I thought we should simply go with DBL, short for Domain Blocklist. After all, we are only importing domains. The new home of the project therefore is https://www.ipfire.org/dbl
>>>>>>>>> 
>>>>>>>>> I have added a couple more lists that I thought interesting and I have added a couple more sources that I considered a good start. Hopefully, we will soon gather some more feedback on how well this is all holding up. My main focus has however been on the technology that will power this project.
>>>>>>>>> 
>>>>>>>>> One of the bigger challenges was to create Suricata rules from the lists. Initially I tried to create a ton of rules but since our lists are so large, this quickly became too complicated. I have now settled on using a feature that is only available in more recent versions of Suricata (I believe 7 and later), but since we are already on Suricata 8 in IPFire this won’t be a problem for us. All domains for each list are basically compiled into one massively large dataset and one single rule is referring to that dataset. This way, we won’t have the option to remove any false-positives, but at least Suricata and the GUI won’t starve a really bad death when loading millions of rules.
>>>>>>>>> 
>>>>>>>>> Suricata will now be able to use our rules to block access to any listed domains of each of the categories over DNS, HTTP, TLS or QUIC. Although I don’t expect many users to use Suricata to block porn or other things, this is a great backstop to enforce any policy like that. For example, if there is a user on the network who is trying to circumvent the DNS server that might filter out certain domains, even after getting an IP address resolved through other means, they won’t be able to open a TLS/QUIC connection or send a HTTP request to all blocked domains. Some people have said they were interested in blocking DNS-over-HTTPS and this is a perfect way to do this and actually be sure that any server that is being blocked on the list will actually be completely inaccessible.
>>>>>>>>> 
>>>>>>>>> Those Suricata rules are already available for testing in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=9eb8751487d23dd354a105c28bdbbb0398fe6e85
>>>>>>>>> 
>>>>>>>>> I have chosen various severities for the lists. If someone was to block advertising using DBL, this is fine, but not a very severe alert. If someone chooses to block malware and there is a system on the network trying to access those domains, this is an alert worth being investigated by an admin. Our new Suricata Reporter will show those violations in different colours based on the severity which helps to identify the right alerts to further investigate.
>>>>>>>>> 
>>>>>>>>> Formerly I have asked you to test the lists using URL Filter. Those rules are now available as well in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=db160694279a4b10378447f775dd536fdfcfb02a
>>>>>>>>> 
>>>>>>>>> I talked about a method to remove any dead domains from any sources which is a great way to keep our lists smaller. The pure size of them is a problem in so many ways. That check was however a little bit too ambitious and I had to make it a little bit less eager. Basically if we are in doubt, we need to still list the domain because it might be resolvable by a user.
>>>>>>>>> 
>>>>>>>>> https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=bb5b6e33b731501d45dea293505f7d42a61d5ce7
>>>>>>>>> 
>>>>>>>>> So how else could we make the lists smaller without losing any actual data? Since we sometimes list a whole TLD (e.g. .xxx or .porn), there is very little point in listing any domains of this TLD. They will always be caught anyways. So I built a check that marks all domains that don’t need to be included on the exported lists because they will never be needed and was able to shrink the size of the lists by a lot again.
>>>>>>>>> 
>>>>>>>>> The website does not show this data, but the API returns the number of “subsumed” domains (I didn’t have a better name):
>>>>>>>>> 
>>>>>>>>> curl https://api.dbl.ipfire.org/lists | jq .
>>>>>>>>> 
>>>>>>>>> The number shown would normally be added to the total number of domains and usually cuts the size of the list by 50-200%.
>>>>>>>>> 
>>>>>>>>> Those stats will now also be stored in a history table so that we will be able to track growth of all lists.
>>>>>>>>> 
>>>>>>>>> Furthermore, the application will now send email notifications for any incoming reports. This way, we will be able to stay in close touch with the reporters and keep them up to date on their submissions as well as inform moderators that there is something to have a look at.
>>>>>>>>> 
>>>>>>>>> The search has been refactored as well, so that we can show clearly whether something is blocked or not at one glance: https://www.ipfire.org/dbl/search?q=github.com. There is detailed information available on all domains and what happened to them. In case of GitHub.com, this seems to be blocked and unblocked by someone all of the time and we can see a clear audit trail of that: https://www.ipfire.org/dbl/lists/malware/domains/github.com
>>>>>>>>> 
>>>>>>>>> On the DNS front, I have added some metadata to the zones so that people can programmatically request some data, like when it has been last updated (in a human-friendly timestamp and not only the serial), license, description and so on:
>>>>>>>>> 
>>>>>>>>> # dig +short ANY _info.ads.dbl.ipfire.org @primary.dbl.ipfire.org
>>>>>>>>> "total-domains=42226"
>>>>>>>>> "license=CC BY-SA 4.0"
>>>>>>>>> "updated-at=2026-01-20T22:17:02.409933+00:00"
>>>>>>>>> "description=Blocks domains used for ads, tracking, and ad delivery”
>>>>>>>>> 
>>>>>>>>> Now, I would like to hear more feedback from you. I know we've all been stretched thin lately, so I especially appreciate anyone who has time to review and provide input. Ideas, just say if you like it or not. Where this could go in the future?
>>>>>>>>> 
>>>>>>>>> Looking ahead, I would like us to start thinking about the RPZ feature that has been on the wishlist. IPFire DBL has been a bigger piece of work, and I think it's worth having a conversation about sustainability. Resources for this need to be allocated and paid for. Open source is about freedom, not free beer — and to keep building features like this, we will need to explore some funding options. I would be interested to hear any ideas you might have that could work for IPFire.
>>>>>>>>> 
>>>>>>>>> Please share your thoughts on the mailing list when you can — even a quick 'looks good' or 'I have concerns about X' is valuable. Public discussion helps everyone stay in the loop and contribute.
>>>>>>>>> 
>>>>>>>>> I am aiming to move forward with this in a week's time, so if you have input, now would be a good time to share it.
>>>>>>>>> 
>>>>>>>>> Best,
>>>>>>>>> -Michael
>>>>>>>>> 
>>>>>>>>>> On 6 Jan 2026, at 10:20, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>>>>>>>>> 
>>>>>>>>>> Good Morning Adolf,
>>>>>>>>>> 
>>>>>>>>>> I had a look at this problem yesterday and it seems that parsing the format is becoming a little bit difficult this way. Since this is only affecting very few domains, I have simply whitelisted them all manually and duckduckgo.com <http://duckduckgo.com/> and others should now be easily reachable again.
>>>>>>>>>> 
>>>>>>>>>> Please let me know if you have any more findings.
>>>>>>>>>> 
>>>>>>>>>> All the best,
>>>>>>>>>> -Michael
>>>>>>>>>> 
>>>>>>>>>>> On 5 Jan 2026, at 11:48, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Hello Adolf,
>>>>>>>>>>> 
>>>>>>>>>>> This is a good find.
>>>>>>>>>>> 
>>>>>>>>>>> But if duckduckgo.com <http://duckduckgo.com/> is blocked, we will have to have a source somewhere that blocks that domain. Not only a sub-domain of it. Otherwise we have a bug somewhere.
>>>>>>>>>>> 
>>>>>>>>>>> This is most likely as the domain is listed here, but with some stuff afterwards:
>>>>>>>>>>> 
>>>>>>>>>>> https://raw.githubusercontent.com/mtxadmin/ublock/refs/heads/master/hosts/_malware_typo
>>>>>>>>>>> 
>>>>>>>>>>> We strip everything after a # away because we consider it a comment. However, that causes that there is only a line with the domain left which will cause it being listed.
>>>>>>>>>>> 
>>>>>>>>>>> The # sign is used as some special character but at the same time it is being used for comments.
>>>>>>>>>>> 
>>>>>>>>>>> I will fix this and then refresh the list.
>>>>>>>>>>> 
>>>>>>>>>>> -Michael
>>>>>>>>>>> 
>>>>>>>>>>>> On 5 Jan 2026, at 11:31, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On 05/01/2026 12:11, Adolf Belka wrote:
>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I have found that the malware list includes duckduckgo.com
>>>>>>>>>>>>> 
>>>>>>>>>>>> I have checked through the various sources used for the malware list.
>>>>>>>>>>>> 
>>>>>>>>>>>> The ShadowWhisperer (Tracking) list has improving.duckduckgo.com in its list. I suspect that this one is the one causing the problem.
>>>>>>>>>>>> 
>>>>>>>>>>>> The mtxadmin (_malware_typo) list has duckduckgo.com mentioned 3 times but not directly as a domain name - looks more like a reference.
>>>>>>>>>>>> 
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> 
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 02/01/2026 14:02, Adolf Belka wrote:
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 02/01/2026 12:09, Michael Tremer wrote:
>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On 30 Dec 2025, at 14:05, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On 29/12/2025 13:05, Michael Tremer wrote:
>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I hope everyone had a great Christmas and a couple of quiet days to relax from all the stress that was the year 2025.
>>>>>>>>>>>>>>>> Still relaxing.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Very good, so let’s have a strong start into 2026 now!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Starting next week, yes.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Having a couple of quieter days, I have been working on a new, little (hopefully) side project that has probably been high up on our radar since the Shalla list has shut down in 2020, or maybe even earlier. The goal of the project is to provide good lists with categories of domain names which are usually used to block access to these domains.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> How did we get here?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> As stated before, the URL filter feature in IPFire has the problem that there are not many good blocklists available any more. There used to be a couple more - most famously the Shalla list - but we are now down to a single list from the University of Toulouse. It is a great list, but it is not always the best fit for all users.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Then there has been talk about whether we could implement more blocking features into IPFire that don’t involve the proxy. Most famously blocking over DNS. The problem here remains a the blocking feature is only as good as the data that is fed into it. Some people have been putting forward a number of lists that were suitable for them, but they would not have replaced the blocking functionality as we know it. Their aim is to provide “one list for everything” but that is not what people usually want. It is targeted at a classic home user and the only separation that is being made is any adult/porn/NSFW content which usually is put into a separate list.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> It would have been technically possible to include these lists and let the users decide, but that is not the aim of IPFire. We want to do the job for the user so that their job is getting easier. Including obscure lists that don’t have a clear outline of what they actually want to block (“bad content” is not a category) and passing the burden of figuring out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with cream on top is really not going to work. It is all confusing and will lead to a bad user experience.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> An even bigger problem that is however completely impossible to solve is bad licensing of these lists. A user has asked the publisher of the HaGeZi list whether they could be included in IPFire and under what terms. The response was that the list is available under the terms of the GNU General Public License v3, but that does not seem to be true. The list contains data from various sources. Many of them are licensed under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public agreement that this data may be redistributed, there is a huge legal issue here. We would expose our users to potential copyright infringement which we cannot do under any circumstances. Furthermore many lists are available under a non-commercial license which excludes them from being used in any kind of business. Plenty of IPFire systems are running in businesses, if not even the vast majority.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> In short, these lists are completely unusable for us. Apart from HaGeZi, I consider OISD to have the same problem.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Enough about all the things that are bad. Let’s talk about the new, good things:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Many blacklists on the internet are an amalgamation of other lists. These lists vary in quality with some of them being not that good and without a clear focus and others being excellent data. Since we don’t have the man power to start from scratch, I felt that we can copy the concept that HaGeZi and OISD have started and simply create a new list that is based on other lists at the beginning to have a good starting point. That way, we have much better control over what is going on these lists and we can shape and mould them as we need them. Most importantly, we don’t create a single lists, but many lists that have a clear focus and allow users to choose what they want to block and what not.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> So the current experimental stage that I am in has these lists:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> * Ads
>>>>>>>>>>>>>>>>> * Dating
>>>>>>>>>>>>>>>>> * DoH
>>>>>>>>>>>>>>>>> * Gambling
>>>>>>>>>>>>>>>>> * Malware
>>>>>>>>>>>>>>>>> * Porn
>>>>>>>>>>>>>>>>> * Social
>>>>>>>>>>>>>>>>> * Violence
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The categories have been determined by what source lists we have available with good data and are compatible with our chosen license CC BY-SA 4.0. This is the same license that we are using for the IPFire Location database, too.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The main use-cases for any kind of blocking are to comply with legal requirements in networks with children (i.e. schools) to remove any kind of pornographic content, sometimes block social media as well. Gambling and violence are commonly blocked, too. Even more common would be filtering advertising and any malicious content.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The latter is especially difficult because so many source lists throw phishing, spyware, malvertising, tracking and other things into the same bucket. Here this is currently all in the malware list which has therefore become quite large. I am not sure whether this will stay like this in the future or if we will have to make some adjustments, but that is exactly why this is now entering some larger testing.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> What has been built so far? In order to put these lists together properly, track any data about where it is coming from, I have built a tool in Python available here:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> https://git.ipfire.org/?p=dnsbl.git;a=summary
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> This tool will automatically update all lists once an hour if there have been any changes and export them in various formats. The exported lists are available for download here:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> https://dnsbl.ipfire.org/lists/
>>>>>>>>>>>>>>>> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the custom url works fine.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> However you need to remember not to put the https:// at the front of the url otherwise the WUI page completes without any error messages but leaves an error message in the system logs saying
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> URL filter blacklist - ERROR: Not a valid URL filter blacklist
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I found this out the hard way.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Oh yes, I forgot that there is a field on the web UI. If that does not accept https:// as a prefix, please file a bug and we will fix it.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I will confirm it and raise a bug.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The other thing I noticed is that if you already have the Toulouse University list downloaded and you then change to the ipfire custom url then all the existing Toulouse blocklists stay in the directory on IPFire and so you end up with a huge number of category tick boxes, most of which are the old Toulouse ones, which are still available to select and it is not clear which ones are from Toulouse and which ones from IPFire.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Yes, I got the same thing, too. I think this is a bug, too, because otherwise you would have a lot of unused categories lying around that will never be updated. You cannot even tell which ones are from the current list and which ones from the old list.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Long-term we could even consider to remove the Univ. Toulouse list entirely and only have our own lists available which would make the problem go away.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I think if the blocklist URL source is changed or a custom url is provided the first step should be to remove the old ones already existing.
>>>>>>>>>>>>>>>> That might be a problem because users can also create their own blocklists and I believe those go into the same directory.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Good thought. We of course cannot delete the custom lists.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Without clearing out the old blocklists you end up with a huge number of checkboxes for lists but it is not clear what happens if there is a category that has the same name for the Toulouse list and the IPFire list such as gambling. I will have a look at that and see what happens.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Not sure what the best approach to this is.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I believe it is removing all old content.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Manually deleting all contents of the urlfilter/blacklists/ directory and then selecting the IPFire blocklist url for the custom url I end up with only the 8 categories from the IPFire list.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I have tested some gambling sites from the IPFire list and the block worked on some. On others the site no longer exists so there is nothing to block or has been changed to an https site and in that case it went straight through. Also if I chose the http version of the link, it was automatically changed to https and went through without being blocked.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The entire IPFire infrastructure always requires HTTPS. If you start using HTTP, you will be automatically redirected. It is 2026 and we don’t need to talk HTTP any more :)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Some of the domains in the gambling list (maybe quite a lot) seem to only have an http access. If I tried https it came back with the fact that it couldn't find it.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I am glad to hear that the list is actually blocking. It would have been bad if it didn’t. Now we have the big task to check out the “quality” - however that can be determined. I think this is what needs some time…
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> In the meantime I have set up a small page on our website:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> https://www.ipfire.org/dnsbl
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I would like to run this as a first-class project inside IPFire like we are doing with IPFire Location. That means that we need to tell people about what we are doing. Hopefully this page is a little start.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Initially it has a couple of high-level bullet points about what we are trying to achieve. I don’t think the text is very good, yet, but it is the best I had in that moment. There is then also a list of the lists that we currently offer. For each list, a detailed page will tell you about the license, how many domains are listed, when the last update has been, the sources and even there is a history page that shows all the changes whenever they have happened.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Finally there is a section that explains “How To Use?” the list which I would love to extend to include AdGuard Plus and things like that as well as Pi-Hole and whatever else could use the list. In a later step we should go ahead and talk to any projects to include our list(s) into their dropdown so that people can enable them nice and easy.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Behind the web page there is an API service that is running on the host that is running the DNSBL. The frontend web app that is running www.ipfire.org <http://www.ipfire.org/> is connecting to that API service to fetch the current lists, any details and so on. That way, we can split the logic and avoid creating a huge monolith of a web app. This also means that page could be down a little as I am still working on the entire thing and will frequently restart it.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The API documentation is available here and the API is publicly available: https://api.dnsbl.ipfire.org/docs
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The website/API allows to file reports for anything that does not seem to be right on any of the lists. I would like to keep it as an open process, however, long-term, this cannot cost us any time. In the current stage, the reports are getting filed and that is about it. I still need to build out some way for admins or moderators (I am not sure what kind of roles I want to have here) to accept or reject those reports.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> In case of us receiving a domain from a source list, I would rather like to submit a report to upstream for them to de-list. That way, we don’t have any admin to do and we are contributing back to other list. That would be a very good thing to do. We cannot however throw tons of emails at some random upstream projects without co-ordinating this first. By not reporting upstream, we will probably over time create large whitelists and I am not sure if that is a good thing to do.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Finally, there is a search box that can be used to find out if a domain is listed on any of the lists.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> If you download and open any of the files, you will see a large header that includes copyright information and lists all sources that have been used to create the individual lists. This way we ensure maximum transparency, comply with the terms of the individual licenses of the source lists and give credit to the people who help us to put together the most perfect list for our users.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I would like this to become a project that is not only being used in IPFire. We can and will be compatible with other solutions like AdGuard, PiHole so that people can use our lists if they would like to even though they are not using IPFire. Hopefully, these users will also feed back to us so that we can improve our lists over time and make them one of the best options out there.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> All lists are available as a simple text file that lists the domains. Then there is a hosts file available as well as a DNS zone file and an RPZ file. Each list is individually available to be used in squidGuard and there is a larger tarball available with all lists that can be used in IPFire’s URL Filter. I am planning to add Suricata/Snort signatures whenever I have time to do so. Even though it is not a good idea to filter pornographic content this way, I suppose that catching malware and blocking DoH are good use-cases for an IPS. Time will tell…
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> As a start, we will make these lists available in IPFire’s URL Filter and collect some feedback about how we are doing. Afterwards, we can see where else we can take this project.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> If you want to enable this on your system, simply add the URL to your autoupdate.urls file like here:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f
>>>>>>>>>>>>>>>> I also tested out adding the IPFire url to autoupdate.urls and that also worked fine for me.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Very good. Should we include this already with Core Update 200? I don’t think we would break anything, but we might already gain a couple more people who are helping us to test this all?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I think that would be a good idea.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The next step would be to build and test our DNS infrastructure. In the “How To Use?” Section on the pages of the individual lists, you can already see some instructions on how to use the lists as an RPZ. In comparison to other “providers”, I would prefer if people would be using DNS to fetch the lists. This is simply to push out updates in a cheap way for us and also do it very regularly.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Initially, clients will pull the entire list using AXFR. There is no way around this as they need to have the data in the first place. After that, clients will only need the changes. As you can see in the history, the lists don’t actually change that often. Sometimes only once a day and therefore downloading the entire list again would be a huge waste of data, both on the client side, but also for us hosting then.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Some other providers update their lists “every 10 minutes”, and there won't be any changes whatsoever. We don’t do that. We will only export the lists again when they have actually changed. The timestamps on the files that we offer using HTTPS can be checked by clients so that they won’t re-download the list again if it has not been changed. But using HTTPS still means that we would have to re-download the entire list and not only the changes.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Using DNS and IXFR will update the lists by only transferring a few kilobytes and therefore we can have clients check once an hour if a list has actually changed and only send out the raw changes. That way, we will be able to serve millions of clients at very cheap cost and they will always have a very up to date list.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> As far as I can see any DNS software that supports RPZs supports AXFR/IXFR with exception of Knot Resolver which expects the zone to be downloaded externally. There is a ticket for AXFR/IXFR support (https://gitlab.nic.cz/knot/knot-resolver/-/issues/195).
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Initially, some of the lists have been *huge* which is why a simple HTTP download is not feasible. The porn list was over 100 MiB. We could have spent thousands on just traffic alone which I don’t have for this kind of project. It would also be unnecessary money being spent. There are simply better solutions out there. But then I built something that basically tests the data that we are receiving from upstream but simply checking if a listed domain still exists. The result was very astonishing to me.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> So whenever someone adds a domain to the list, we will (eventually, but not immediately) check if we can resolve the domain’s SOA record. If not, we mark the domain as non-active and will no longer include them in the exported data. This brought down the porn list from just under 5 million domains to just 421k. On the sources page (https://www.ipfire.org/dnsbl/lists/porn/sources) I am listing the percentage of dead domains from each of them and the UT1 list has 94% dead domains. Wow.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> If we cannot resolve the domain, neither can our users. So we would otherwise fill the lists with tons of domains that simply could never be reached. And if they cannot be reached, why would we block them? We would waste bandwidth and a lot of memory on each single client.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The other sources have similarly high rations of dead domains. Most of them are in the 50-80% range. Therefore I am happy that we are doing some extra work here to give our users much better data for their filtering.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Removing all dead entries sounds like an excellent step.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> So, if you like, please go and check out the RPZ blocking with Unbound. Instructions are on the page. I would be happy to hear how this is turning out.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Please let me know if there are any more questions, and I would be glad to answer them.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Happy New Year,
>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>> This email is just a brain dump from me to this list. I would be happy to answer any questions about implementation details, etc. if people are interested. Right now, this email is long enough already…
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> All the best,
>>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> -- 
>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> <suricata-log.jpg><ipfire_dnsbl-violence.rules>
>>> 
>> 
> 

Re: Let's launch our own blocklists...

[Michael Tremer]

Hello Matthias,

> On 28 Jan 2026, at 21:25, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
> 
> Hi,
> 
> On 28.01.2026 17:59, Michael Tremer wrote:
>> Thank you for the feedback. Glad to hear that the changes are working well.
> 
> They are working a bit too good...

Can there be too good?

> I had to deactivate the TLS and HTTP Rules in 'Advertising' and
> 'Malware' because my proxy got blocked:

Yeah, this is what I pointed out in an earlier email. If you block in the IPS and elsewhere (i.e. URL Filter), this will become confusing. Error pages are not being shown, and the IPS is just killing connections.

But it does work as designed…

-Michael

> ***SNIP***
> ...
> Date: 01/28 17:45:39
> Name: IPFire DBL [Malware] Blocked HTTP Request
> Priority: 1
> Type: Potential Corporate Privacy Violation
> IP Info: 192.168.XXX.YYY:50132 -> 192.168.XXX.YYY:8080
> SID: 786434
> Refs:
> ...
> Date: 01/28 17:45:39
> Name: IPFire DBL [Advertising] Blocked HTTP Request
> Priority: 3
> Type: Potential Corporate Privacy Violation
> IP Info: 192.168.XXX.YYY:50133 -> 192.168.XXX.YYY:8080
> SID: 983042
> Refs:
> ...
> Date: 01/28 18:55:01
> Name: IPFire DBL [Advertising] Blocked TLS Connection
> Priority: 3
> Type: Potential Corporate Privacy Violation
> IP Info: 192.168.XXX.YYY:51061 -> 192.168.XXX.YYY:8080
> SID: 983043
> Refs:
> ...
> Date: 01/28 21:15:47
> Name: IPFire DBL [Malware] Blocked TLS Connection
> Priority: 1
> Type: Potential Corporate Privacy Violation
> IP Info: 192.168.XXX.YYY:51766 -> 192.168.XXX.YYY:8080
> SID: 786435
> Refs:
> ...
> ***SNAP***
> 
> Best
> Matthias
> 
>>> On 28 Jan 2026, at 16:33, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>>> 
>>> On 26.01.2026 18:18, Michael Tremer wrote:
>>>> Hello Matthias,
>>> 
>>> Hi,
>>> 
>>>> Are you running this on Core Update 200?
>>> 
>>> Now running on Core 199 - after applying...
>>> 
>>>> There were some changes required so that we will extract the datasets from the rules tarballs. I am using a special feature in Suricata so that the lists won’t be using too much memory and will be quickly searchable:
>>>> 
>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=f0b43241a501f7c545e3cb15f6989e945c60b3e2
>>> 
>>> ...these patches.
>>> 
>>> No more errors. ;-)
>>> 
>>> Best
>>> Matthias
>>> 
>>>> -Michael
>>>> 
>>>>> On 25 Jan 2026, at 17:50, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>>>>> 
>>>>> On 25.01.2026 15:40, Michael Tremer wrote:
>>>>>> Hello Matthias,
>>>>> 
>>>>> Hi Michael,
>>>>> 
>>>>>> Nice catch!
>>>>>> 
>>>>>> I fixed it here and added the missing “;”:
>>>>> 
>>>>> Yep. The missing ";" seems to be fixed, but 'suricata' still doesn't
>>>>> like our rules. I added 'violence' as an attachment... ;-)
>>>>> 
>>>>>> https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=775561e322ceed43e255e5547bd76047b9f8a40b
>>>>>> 
>>>>>> If you go to the provider settings there is a button to force a ruleset update which should give you the fixed version. Please let me know if this works.
>>>>> 
>>>>> I did that - rules were updated, but no luck:
>>>>> 
>>>>> ***SNIP***
>>>>> ...
>>>>> 18:37:37  suricata:  [2577] <Info> -- Including configuration file
>>>>> /var/ipfire/suricata/suricata-used-rulesfiles.yaml.
>>>>> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
>>>>> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop dns
>>>>> any any -> any any (msg:"IPFire DBL [Violence] Blocked DNS Query";
>>>>> dns.query; domain; dataset:isset,violence,type string,load
>>>>> datasets/violence.txt; classtype:policy-violation; priority:2;
>>>>> sid:1048577; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
>>>>> metadata:dbl violence.dbl.ipfire.org;)" from file
>>>>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 39
>>>>> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
>>>>> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop
>>>>> http any any -> any any (msg:"IPFire DBL [Violence] Blocked HTTP
>>>>> Request"; http.host; domain; dataset:isset,violence,type string,load
>>>>> datasets/violence.txt; classtype:policy-violation; priority:2;
>>>>> sid:1048578; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
>>>>> metadata:dbl violence.dbl.ipfire.org;)" from file
>>>>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 40
>>>>> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
>>>>> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop tls
>>>>> any any -> any any (msg:"IPFire DBL [Violence] Blocked TLS Connection";
>>>>> tls.sni; domain; dataset:isset,violence,type string,load
>>>>> datasets/violence.txt; classtype:policy-violation; priority:2;
>>>>> sid:1048579; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
>>>>> metadata:dbl violence.dbl.ipfire.org;)" from file
>>>>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 41
>>>>> 18:37:37  suricata:  [2577] <Error> -- failed to set up dataset 'violence'.
>>>>> 18:37:37  suricata:  [2577] <Error> -- error parsing signature "drop
>>>>> quic any any -> any any (msg:"IPFire DBL [Violence] Blocked QUIC
>>>>> Connection"; quic.sni; domain; dataset:isset,violence,type string,load
>>>>> datasets/violence.txt; classtype:policy-violation; priority:2;
>>>>> sid:1048580; rev:1; reference:url,https://www.ipfire.org/dbl/violence;
>>>>> metadata:dbl violence.dbl.ipfire.org;)" from file
>>>>> /var/lib/suricata/ipfire_dnsbl-violence.rules at line 42
>>>>> ...
>>>>> ***SNAP***
>>>>> 
>>>>> For better reading - see attached screenshot.
>>>>> 
>>>>> Best
>>>>> Matthias
>>>>> 
>>>>>> Best,
>>>>>> -Michael
>>>>>> 
>>>>>>> On 24 Jan 2026, at 23:41, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>>>>>>> 
>>>>>>> On 23.01.2026 17:39, Michael Tremer wrote:
>>>>>>>> Hello Matthias,
>>>>>>> 
>>>>>>> Hi Michael,
>>>>>>> 
>>>>>>>> Thank you very much for testing IPFire DBL.
>>>>>>> 
>>>>>>> No problem - I have news:
>>>>>>> 
>>>>>>> After taking a closer look to the IPS system logs, unfortunately I found
>>>>>>> some parsing errors:
>>>>>>> 
>>>>>>> 'suricata' complains about missing ";".
>>>>>>> 
>>>>>>> ***SNIP***
>>>>>>> ...
>>>>>>> 00:32:40 suricata: [13343] <Info> -- Including configuration file
>>>>>>> /var/ipfire/suricata/suricata-used-rulesfiles.yaml.
>>>>>>> 00:32:40 suricata: [13343] <Error> -- no terminating ";" found
>>>>>>> 00:32:40 suricata: [13343] <Error> -- error parsing signature "drop
>>>>>>> dns any any -> any any (msg:"IPFire DBL [Advertising] Blocked DNS
>>>>>>> Query"; dns.query; domain; dataset:isset,ads,type string,load
>>>>>>> datasets/ads.txt; classtype:policy-violation; priority:3; sid:983041;
>>>>>>> rev:1; reference:url,https://www.ipfire.org/dbl/ads; metadata:dbl
>>>>>>> ads.dbl.ipfire.org)" from file /var/lib/suricata/ipfire_dnsbl-ads.rules
>>>>>>> at line 72
>>>>>>> 00:32:40 suricata: [13343] <Error> -- no terminating ";" found
>>>>>>> ...
>>>>>>> ***SNAP***
>>>>>>> 
>>>>>>> I tried, but didn't find the right place for any missing ";".
>>>>>>> 
>>>>>>> Can "anyone" confirm?
>>>>>>> 
>>>>>>> Best
>>>>>>> Matthias
>>>>>>> 
>>>>>>>>> On 23 Jan 2026, at 15:02, Matthias Fischer <matthias.fischer@ipfire.org> wrote:
>>>>>>>>> 
>>>>>>>>> On 22.01.2026 12:33, Michael Tremer wrote:
>>>>>>>>>> Hello everyone,
>>>>>>>>> 
>>>>>>>>> Hi,
>>>>>>>>> 
>>>>>>>>> short feedback from me:
>>>>>>>>> 
>>>>>>>>> - I activated both the suricata (IPFire DBL - Domain Blocklist) - and
>>>>>>>>> the URLfilter lists from 'dbl.ipfire.org'.
>>>>>>>> 
>>>>>>>> This is an interesting case. What I didn’t manage to test yet is what happens when Suricata blocks the connection first. If URL Filter sees a domain that is being blocked it will either send you an error page if you are using HTTP, or simply close the connection if it is HTTPS. However, when Suricata comes first in the chain (and it will), it might close the connection because URL Filter has received the request. In the case of HTTPS this does not make any difference because the connection will be closed, but in the HTTP case you won’t see an error page any more and instead have the connection closed, too. You are basically losing the explicit error notification which is a little bit annoying.
>>>>>>>> 
>>>>>>>> We could have the same when we are doing the same with Unbound and DNS filtering. Potentially we would need to whitelist the local DNS resolver then, but how is Suricata supposed to know that the same categories are activated in both places?
>>>>>>>> 
>>>>>>>>> - I even took the 'smart-tv' domains from the IFire DBL blacklist and
>>>>>>>>> copied/pasted them in my fritzbox filter lists.
>>>>>>>> 
>>>>>>>> LOL Why not use IPFire to filter this as well?
>>>>>>>> 
>>>>>>>>> Everything works as expected. Besides, the download of the IPFire
>>>>>>>>> DBL-list loads a lot faster than the list from 'Univ. Toulouse'... ;-)
>>>>>>>> 
>>>>>>>> Yes, we don’t have much traffic on the server, yet.
>>>>>>>> 
>>>>>>>>> Functionality is good - no false positives or seen problems. Good work -
>>>>>>>>> thanks!
>>>>>>>> 
>>>>>>>> Nice. We need to distinguish a little between what is a technical issue and what is a false-positive/missing domain on the list. However, testing both at the same time is something we will all cope quite well with :)
>>>>>>>> 
>>>>>>>> -Michael
>>>>>>>> 
>>>>>>>>> Best
>>>>>>>>> Matthias
>>>>>>>>> 
>>>>>>>>>> Over the past few weeks I have made significant progress on this all, and I think we're getting close to something the community will be really happy with. I'd love to get feedback from the team before we finalise things.
>>>>>>>>>> 
>>>>>>>>>> So what has happened?
>>>>>>>>>> 
>>>>>>>>>> First of all, the entire project has been renamed. DNSBL is not entirely what this is. Although the lists can be thrown into DNS, they have much more use outside of it that I thought we should simply go with DBL, short for Domain Blocklist. After all, we are only importing domains. The new home of the project therefore is https://www.ipfire.org/dbl
>>>>>>>>>> 
>>>>>>>>>> I have added a couple more lists that I thought interesting and I have added a couple more sources that I considered a good start. Hopefully, we will soon gather some more feedback on how well this is all holding up. My main focus has however been on the technology that will power this project.
>>>>>>>>>> 
>>>>>>>>>> One of the bigger challenges was to create Suricata rules from the lists. Initially I tried to create a ton of rules but since our lists are so large, this quickly became too complicated. I have now settled on using a feature that is only available in more recent versions of Suricata (I believe 7 and later), but since we are already on Suricata 8 in IPFire this won’t be a problem for us. All domains for each list are basically compiled into one massively large dataset and one single rule is referring to that dataset. This way, we won’t have the option to remove any false-positives, but at least Suricata and the GUI won’t starve a really bad death when loading millions of rules.
>>>>>>>>>> 
>>>>>>>>>> Suricata will now be able to use our rules to block access to any listed domains of each of the categories over DNS, HTTP, TLS or QUIC. Although I don’t expect many users to use Suricata to block porn or other things, this is a great backstop to enforce any policy like that. For example, if there is a user on the network who is trying to circumvent the DNS server that might filter out certain domains, even after getting an IP address resolved through other means, they won’t be able to open a TLS/QUIC connection or send a HTTP request to all blocked domains. Some people have said they were interested in blocking DNS-over-HTTPS and this is a perfect way to do this and actually be sure that any server that is being blocked on the list will actually be completely inaccessible.
>>>>>>>>>> 
>>>>>>>>>> Those Suricata rules are already available for testing in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=9eb8751487d23dd354a105c28bdbbb0398fe6e85
>>>>>>>>>> 
>>>>>>>>>> I have chosen various severities for the lists. If someone was to block advertising using DBL, this is fine, but not a very severe alert. If someone chooses to block malware and there is a system on the network trying to access those domains, this is an alert worth being investigated by an admin. Our new Suricata Reporter will show those violations in different colours based on the severity which helps to identify the right alerts to further investigate.
>>>>>>>>>> 
>>>>>>>>>> Formerly I have asked you to test the lists using URL Filter. Those rules are now available as well in Core Update 200: https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=db160694279a4b10378447f775dd536fdfcfb02a
>>>>>>>>>> 
>>>>>>>>>> I talked about a method to remove any dead domains from any sources which is a great way to keep our lists smaller. The pure size of them is a problem in so many ways. That check was however a little bit too ambitious and I had to make it a little bit less eager. Basically if we are in doubt, we need to still list the domain because it might be resolvable by a user.
>>>>>>>>>> 
>>>>>>>>>> https://git.ipfire.org/?p=dbl.git;a=commitdiff;h=bb5b6e33b731501d45dea293505f7d42a61d5ce7
>>>>>>>>>> 
>>>>>>>>>> So how else could we make the lists smaller without losing any actual data? Since we sometimes list a whole TLD (e.g. .xxx or .porn), there is very little point in listing any domains of this TLD. They will always be caught anyways. So I built a check that marks all domains that don’t need to be included on the exported lists because they will never be needed and was able to shrink the size of the lists by a lot again.
>>>>>>>>>> 
>>>>>>>>>> The website does not show this data, but the API returns the number of “subsumed” domains (I didn’t have a better name):
>>>>>>>>>> 
>>>>>>>>>> curl https://api.dbl.ipfire.org/lists | jq .
>>>>>>>>>> 
>>>>>>>>>> The number shown would normally be added to the total number of domains and usually cuts the size of the list by 50-200%.
>>>>>>>>>> 
>>>>>>>>>> Those stats will now also be stored in a history table so that we will be able to track growth of all lists.
>>>>>>>>>> 
>>>>>>>>>> Furthermore, the application will now send email notifications for any incoming reports. This way, we will be able to stay in close touch with the reporters and keep them up to date on their submissions as well as inform moderators that there is something to have a look at.
>>>>>>>>>> 
>>>>>>>>>> The search has been refactored as well, so that we can show clearly whether something is blocked or not at one glance: https://www.ipfire.org/dbl/search?q=github.com. There is detailed information available on all domains and what happened to them. In case of GitHub.com, this seems to be blocked and unblocked by someone all of the time and we can see a clear audit trail of that: https://www.ipfire.org/dbl/lists/malware/domains/github.com
>>>>>>>>>> 
>>>>>>>>>> On the DNS front, I have added some metadata to the zones so that people can programmatically request some data, like when it has been last updated (in a human-friendly timestamp and not only the serial), license, description and so on:
>>>>>>>>>> 
>>>>>>>>>> # dig +short ANY _info.ads.dbl.ipfire.org @primary.dbl.ipfire.org
>>>>>>>>>> "total-domains=42226"
>>>>>>>>>> "license=CC BY-SA 4.0"
>>>>>>>>>> "updated-at=2026-01-20T22:17:02.409933+00:00"
>>>>>>>>>> "description=Blocks domains used for ads, tracking, and ad delivery”
>>>>>>>>>> 
>>>>>>>>>> Now, I would like to hear more feedback from you. I know we've all been stretched thin lately, so I especially appreciate anyone who has time to review and provide input. Ideas, just say if you like it or not. Where this could go in the future?
>>>>>>>>>> 
>>>>>>>>>> Looking ahead, I would like us to start thinking about the RPZ feature that has been on the wishlist. IPFire DBL has been a bigger piece of work, and I think it's worth having a conversation about sustainability. Resources for this need to be allocated and paid for. Open source is about freedom, not free beer — and to keep building features like this, we will need to explore some funding options. I would be interested to hear any ideas you might have that could work for IPFire.
>>>>>>>>>> 
>>>>>>>>>> Please share your thoughts on the mailing list when you can — even a quick 'looks good' or 'I have concerns about X' is valuable. Public discussion helps everyone stay in the loop and contribute.
>>>>>>>>>> 
>>>>>>>>>> I am aiming to move forward with this in a week's time, so if you have input, now would be a good time to share it.
>>>>>>>>>> 
>>>>>>>>>> Best,
>>>>>>>>>> -Michael
>>>>>>>>>> 
>>>>>>>>>>> On 6 Jan 2026, at 10:20, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Good Morning Adolf,
>>>>>>>>>>> 
>>>>>>>>>>> I had a look at this problem yesterday and it seems that parsing the format is becoming a little bit difficult this way. Since this is only affecting very few domains, I have simply whitelisted them all manually and duckduckgo.com <http://duckduckgo.com/> and others should now be easily reachable again.
>>>>>>>>>>> 
>>>>>>>>>>> Please let me know if you have any more findings.
>>>>>>>>>>> 
>>>>>>>>>>> All the best,
>>>>>>>>>>> -Michael
>>>>>>>>>>> 
>>>>>>>>>>>> On 5 Jan 2026, at 11:48, Michael Tremer <michael.tremer@ipfire.org> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hello Adolf,
>>>>>>>>>>>> 
>>>>>>>>>>>> This is a good find.
>>>>>>>>>>>> 
>>>>>>>>>>>> But if duckduckgo.com <http://duckduckgo.com/> is blocked, we will have to have a source somewhere that blocks that domain. Not only a sub-domain of it. Otherwise we have a bug somewhere.
>>>>>>>>>>>> 
>>>>>>>>>>>> This is most likely as the domain is listed here, but with some stuff afterwards:
>>>>>>>>>>>> 
>>>>>>>>>>>> https://raw.githubusercontent.com/mtxadmin/ublock/refs/heads/master/hosts/_malware_typo
>>>>>>>>>>>> 
>>>>>>>>>>>> We strip everything after a # away because we consider it a comment. However, that causes that there is only a line with the domain left which will cause it being listed.
>>>>>>>>>>>> 
>>>>>>>>>>>> The # sign is used as some special character but at the same time it is being used for comments.
>>>>>>>>>>>> 
>>>>>>>>>>>> I will fix this and then refresh the list.
>>>>>>>>>>>> 
>>>>>>>>>>>> -Michael
>>>>>>>>>>>> 
>>>>>>>>>>>>> On 5 Jan 2026, at 11:31, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 05/01/2026 12:11, Adolf Belka wrote:
>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I have found that the malware list includes duckduckgo.com
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> I have checked through the various sources used for the malware list.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The ShadowWhisperer (Tracking) list has improving.duckduckgo.com in its list. I suspect that this one is the one causing the problem.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The mtxadmin (_malware_typo) list has duckduckgo.com mentioned 3 times but not directly as a domain name - looks more like a reference.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 02/01/2026 14:02, Adolf Belka wrote:
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 02/01/2026 12:09, Michael Tremer wrote:
>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 30 Dec 2025, at 14:05, Adolf Belka <adolf.belka@ipfire.org> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 29/12/2025 13:05, Michael Tremer wrote:
>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I hope everyone had a great Christmas and a couple of quiet days to relax from all the stress that was the year 2025.
>>>>>>>>>>>>>>>>> Still relaxing.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Very good, so let’s have a strong start into 2026 now!
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Starting next week, yes.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Having a couple of quieter days, I have been working on a new, little (hopefully) side project that has probably been high up on our radar since the Shalla list has shut down in 2020, or maybe even earlier. The goal of the project is to provide good lists with categories of domain names which are usually used to block access to these domains.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> How did we get here?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> As stated before, the URL filter feature in IPFire has the problem that there are not many good blocklists available any more. There used to be a couple more - most famously the Shalla list - but we are now down to a single list from the University of Toulouse. It is a great list, but it is not always the best fit for all users.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Then there has been talk about whether we could implement more blocking features into IPFire that don’t involve the proxy. Most famously blocking over DNS. The problem here remains a the blocking feature is only as good as the data that is fed into it. Some people have been putting forward a number of lists that were suitable for them, but they would not have replaced the blocking functionality as we know it. Their aim is to provide “one list for everything” but that is not what people usually want. It is targeted at a classic home user and the only separation that is being made is any adult/porn/NSFW content which usually is put into a separate list.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> It would have been technically possible to include these lists and let the users decide, but that is not the aim of IPFire. We want to do the job for the user so that their job is getting easier. Including obscure lists that don’t have a clear outline of what they actually want to block (“bad content” is not a category) and passing the burden of figuring out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with cream on top is really not going to work. It is all confusing and will lead to a bad user experience.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> An even bigger problem that is however completely impossible to solve is bad licensing of these lists. A user has asked the publisher of the HaGeZi list whether they could be included in IPFire and under what terms. The response was that the list is available under the terms of the GNU General Public License v3, but that does not seem to be true. The list contains data from various sources. Many of them are licensed under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public agreement that this data may be redistributed, there is a huge legal issue here. We would expose our users to potential copyright infringement which we cannot do under any circumstances. Furthermore many lists are available under a non-commercial license which excludes them from being used in any kind of business. Plenty of IPFire systems are running in businesses, if not even the vast majority.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> In short, these lists are completely unusable for us. Apart from HaGeZi, I consider OISD to have the same problem.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Enough about all the things that are bad. Let’s talk about the new, good things:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Many blacklists on the internet are an amalgamation of other lists. These lists vary in quality with some of them being not that good and without a clear focus and others being excellent data. Since we don’t have the man power to start from scratch, I felt that we can copy the concept that HaGeZi and OISD have started and simply create a new list that is based on other lists at the beginning to have a good starting point. That way, we have much better control over what is going on these lists and we can shape and mould them as we need them. Most importantly, we don’t create a single lists, but many lists that have a clear focus and allow users to choose what they want to block and what not.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> So the current experimental stage that I am in has these lists:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> * Ads
>>>>>>>>>>>>>>>>>> * Dating
>>>>>>>>>>>>>>>>>> * DoH
>>>>>>>>>>>>>>>>>> * Gambling
>>>>>>>>>>>>>>>>>> * Malware
>>>>>>>>>>>>>>>>>> * Porn
>>>>>>>>>>>>>>>>>> * Social
>>>>>>>>>>>>>>>>>> * Violence
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> The categories have been determined by what source lists we have available with good data and are compatible with our chosen license CC BY-SA 4.0. This is the same license that we are using for the IPFire Location database, too.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> The main use-cases for any kind of blocking are to comply with legal requirements in networks with children (i.e. schools) to remove any kind of pornographic content, sometimes block social media as well. Gambling and violence are commonly blocked, too. Even more common would be filtering advertising and any malicious content.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> The latter is especially difficult because so many source lists throw phishing, spyware, malvertising, tracking and other things into the same bucket. Here this is currently all in the malware list which has therefore become quite large. I am not sure whether this will stay like this in the future or if we will have to make some adjustments, but that is exactly why this is now entering some larger testing.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> What has been built so far? In order to put these lists together properly, track any data about where it is coming from, I have built a tool in Python available here:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> https://git.ipfire.org/?p=dnsbl.git;a=summary
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> This tool will automatically update all lists once an hour if there have been any changes and export them in various formats. The exported lists are available for download here:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> https://dnsbl.ipfire.org/lists/
>>>>>>>>>>>>>>>>> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the custom url works fine.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> However you need to remember not to put the https:// at the front of the url otherwise the WUI page completes without any error messages but leaves an error message in the system logs saying
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> URL filter blacklist - ERROR: Not a valid URL filter blacklist
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I found this out the hard way.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Oh yes, I forgot that there is a field on the web UI. If that does not accept https:// as a prefix, please file a bug and we will fix it.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I will confirm it and raise a bug.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The other thing I noticed is that if you already have the Toulouse University list downloaded and you then change to the ipfire custom url then all the existing Toulouse blocklists stay in the directory on IPFire and so you end up with a huge number of category tick boxes, most of which are the old Toulouse ones, which are still available to select and it is not clear which ones are from Toulouse and which ones from IPFire.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Yes, I got the same thing, too. I think this is a bug, too, because otherwise you would have a lot of unused categories lying around that will never be updated. You cannot even tell which ones are from the current list and which ones from the old list.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Long-term we could even consider to remove the Univ. Toulouse list entirely and only have our own lists available which would make the problem go away.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I think if the blocklist URL source is changed or a custom url is provided the first step should be to remove the old ones already existing.
>>>>>>>>>>>>>>>>> That might be a problem because users can also create their own blocklists and I believe those go into the same directory.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Good thought. We of course cannot delete the custom lists.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Without clearing out the old blocklists you end up with a huge number of checkboxes for lists but it is not clear what happens if there is a category that has the same name for the Toulouse list and the IPFire list such as gambling. I will have a look at that and see what happens.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Not sure what the best approach to this is.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I believe it is removing all old content.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Manually deleting all contents of the urlfilter/blacklists/ directory and then selecting the IPFire blocklist url for the custom url I end up with only the 8 categories from the IPFire list.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I have tested some gambling sites from the IPFire list and the block worked on some. On others the site no longer exists so there is nothing to block or has been changed to an https site and in that case it went straight through. Also if I chose the http version of the link, it was automatically changed to https and went through without being blocked.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The entire IPFire infrastructure always requires HTTPS. If you start using HTTP, you will be automatically redirected. It is 2026 and we don’t need to talk HTTP any more :)
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Some of the domains in the gambling list (maybe quite a lot) seem to only have an http access. If I tried https it came back with the fact that it couldn't find it.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I am glad to hear that the list is actually blocking. It would have been bad if it didn’t. Now we have the big task to check out the “quality” - however that can be determined. I think this is what needs some time…
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> In the meantime I have set up a small page on our website:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> https://www.ipfire.org/dnsbl
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I would like to run this as a first-class project inside IPFire like we are doing with IPFire Location. That means that we need to tell people about what we are doing. Hopefully this page is a little start.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Initially it has a couple of high-level bullet points about what we are trying to achieve. I don’t think the text is very good, yet, but it is the best I had in that moment. There is then also a list of the lists that we currently offer. For each list, a detailed page will tell you about the license, how many domains are listed, when the last update has been, the sources and even there is a history page that shows all the changes whenever they have happened.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Finally there is a section that explains “How To Use?” the list which I would love to extend to include AdGuard Plus and things like that as well as Pi-Hole and whatever else could use the list. In a later step we should go ahead and talk to any projects to include our list(s) into their dropdown so that people can enable them nice and easy.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Behind the web page there is an API service that is running on the host that is running the DNSBL. The frontend web app that is running www.ipfire.org <http://www.ipfire.org/> is connecting to that API service to fetch the current lists, any details and so on. That way, we can split the logic and avoid creating a huge monolith of a web app. This also means that page could be down a little as I am still working on the entire thing and will frequently restart it.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The API documentation is available here and the API is publicly available: https://api.dnsbl.ipfire.org/docs
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The website/API allows to file reports for anything that does not seem to be right on any of the lists. I would like to keep it as an open process, however, long-term, this cannot cost us any time. In the current stage, the reports are getting filed and that is about it. I still need to build out some way for admins or moderators (I am not sure what kind of roles I want to have here) to accept or reject those reports.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> In case of us receiving a domain from a source list, I would rather like to submit a report to upstream for them to de-list. That way, we don’t have any admin to do and we are contributing back to other list. That would be a very good thing to do. We cannot however throw tons of emails at some random upstream projects without co-ordinating this first. By not reporting upstream, we will probably over time create large whitelists and I am not sure if that is a good thing to do.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Finally, there is a search box that can be used to find out if a domain is listed on any of the lists.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> If you download and open any of the files, you will see a large header that includes copyright information and lists all sources that have been used to create the individual lists. This way we ensure maximum transparency, comply with the terms of the individual licenses of the source lists and give credit to the people who help us to put together the most perfect list for our users.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I would like this to become a project that is not only being used in IPFire. We can and will be compatible with other solutions like AdGuard, PiHole so that people can use our lists if they would like to even though they are not using IPFire. Hopefully, these users will also feed back to us so that we can improve our lists over time and make them one of the best options out there.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> All lists are available as a simple text file that lists the domains. Then there is a hosts file available as well as a DNS zone file and an RPZ file. Each list is individually available to be used in squidGuard and there is a larger tarball available with all lists that can be used in IPFire’s URL Filter. I am planning to add Suricata/Snort signatures whenever I have time to do so. Even though it is not a good idea to filter pornographic content this way, I suppose that catching malware and blocking DoH are good use-cases for an IPS. Time will tell…
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> As a start, we will make these lists available in IPFire’s URL Filter and collect some feedback about how we are doing. Afterwards, we can see where else we can take this project.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> If you want to enable this on your system, simply add the URL to your autoupdate.urls file like here:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f
>>>>>>>>>>>>>>>>> I also tested out adding the IPFire url to autoupdate.urls and that also worked fine for me.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Very good. Should we include this already with Core Update 200? I don’t think we would break anything, but we might already gain a couple more people who are helping us to test this all?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I think that would be a good idea.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The next step would be to build and test our DNS infrastructure. In the “How To Use?” Section on the pages of the individual lists, you can already see some instructions on how to use the lists as an RPZ. In comparison to other “providers”, I would prefer if people would be using DNS to fetch the lists. This is simply to push out updates in a cheap way for us and also do it very regularly.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Initially, clients will pull the entire list using AXFR. There is no way around this as they need to have the data in the first place. After that, clients will only need the changes. As you can see in the history, the lists don’t actually change that often. Sometimes only once a day and therefore downloading the entire list again would be a huge waste of data, both on the client side, but also for us hosting then.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Some other providers update their lists “every 10 minutes”, and there won't be any changes whatsoever. We don’t do that. We will only export the lists again when they have actually changed. The timestamps on the files that we offer using HTTPS can be checked by clients so that they won’t re-download the list again if it has not been changed. But using HTTPS still means that we would have to re-download the entire list and not only the changes.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Using DNS and IXFR will update the lists by only transferring a few kilobytes and therefore we can have clients check once an hour if a list has actually changed and only send out the raw changes. That way, we will be able to serve millions of clients at very cheap cost and they will always have a very up to date list.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> As far as I can see any DNS software that supports RPZs supports AXFR/IXFR with exception of Knot Resolver which expects the zone to be downloaded externally. There is a ticket for AXFR/IXFR support (https://gitlab.nic.cz/knot/knot-resolver/-/issues/195).
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Initially, some of the lists have been *huge* which is why a simple HTTP download is not feasible. The porn list was over 100 MiB. We could have spent thousands on just traffic alone which I don’t have for this kind of project. It would also be unnecessary money being spent. There are simply better solutions out there. But then I built something that basically tests the data that we are receiving from upstream but simply checking if a listed domain still exists. The result was very astonishing to me.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> So whenever someone adds a domain to the list, we will (eventually, but not immediately) check if we can resolve the domain’s SOA record. If not, we mark the domain as non-active and will no longer include them in the exported data. This brought down the porn list from just under 5 million domains to just 421k. On the sources page (https://www.ipfire.org/dnsbl/lists/porn/sources) I am listing the percentage of dead domains from each of them and the UT1 list has 94% dead domains. Wow.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> If we cannot resolve the domain, neither can our users. So we would otherwise fill the lists with tons of domains that simply could never be reached. And if they cannot be reached, why would we block them? We would waste bandwidth and a lot of memory on each single client.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The other sources have similarly high rations of dead domains. Most of them are in the 50-80% range. Therefore I am happy that we are doing some extra work here to give our users much better data for their filtering.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Removing all dead entries sounds like an excellent step.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> So, if you like, please go and check out the RPZ blocking with Unbound. Instructions are on the page. I would be happy to hear how this is turning out.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Please let me know if there are any more questions, and I would be glad to answer them.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Happy New Year,
>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>> This email is just a brain dump from me to this list. I would be happy to answer any questions about implementation details, etc. if people are interested. Right now, this email is long enough already…
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> All the best,
>>>>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -- 
>>>>>>>>>>>>> Sent from my laptop
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> <suricata-log.jpg><ipfire_dnsbl-violence.rules>
>>>> 
>>> 
>> 
> 

Re: Let's launch our own blocklists...

[Michael Tremer]

Hello Jon,

I cannot see what is different in this email. It looks the same than the previous one to me.

-Michael

> On 30 Dec 2025, at 15:52, Jon Murphy <jon.murphy@ipfire.org> wrote:
> 
> Forgot this part…
> 
> Comments are below.
> 
> 
> ------ Original Message ------
> From "Adolf Belka" <adolf.belka@ipfire.org>
> To "Michael Tremer" <michael.tremer@ipfire.org>
> Cc "IPFire: Development-List" <development@lists.ipfire.org>
> Date 12/30/2025 8:05:20 AM
> Subject Re: Let's launch our own blocklists...
> 
>> Hi Michael,
>> 
>> On 29/12/2025 13:05, Michael Tremer wrote:
>>> Hello everyone,
>>> 
>>> I hope everyone had a great Christmas and a couple of quiet days to relax from all the stress that was the year 2025.
>> Still relaxing.
>>> Having a couple of quieter days, I have been working on a new, little (hopefully) side project that has probably been high up on our radar since the Shalla list has shut down in 2020, or maybe even earlier. The goal of the project is to provide good lists with categories of domain names which are usually used to block access to these domains.
>>> 
>>> I simply call this IPFire DNSBL which is short for IPFire DNS Blocklists.
>>> 
>>> How did we get here?
>>> 
>>> As stated before, the URL filter feature in IPFire has the problem that there are not many good blocklists available any more. There used to be a couple more - most famously the Shalla list - but we are now down to a single list from the University of Toulouse. It is a great list, but it is not always the best fit for all users.
>>> 
>>> Then there has been talk about whether we could implement more blocking features into IPFire that don’t involve the proxy. Most famously blocking over DNS. The problem here remains a the blocking feature is only as good as the data that is fed into it. Some people have been putting forward a number of lists that were suitable for them, but they would not have replaced the blocking functionality as we know it. Their aim is to provide “one list for everything” but that is not what people usually want. It is targeted at a classic home user and the only separation that is being made is any adult/porn/NSFW content which usually is put into a separate list.
>>> 
>>> It would have been technically possible to include these lists and let the users decide, but that is not the aim of IPFire. We want to do the job for the user so that their job is getting easier. Including obscure lists that don’t have a clear outline of what they actually want to block (“bad content” is not a category) and passing the burden of figuring out whether they need the “Light”, “Normal”, “Pro”, “Pro++”, “Ultimate” or even a “Venti” list with cream on top is really not going to work. It is all confusing and will lead to a bad user experience.
>>> 
>>> An even bigger problem that is however completely impossible to solve is bad licensing of these lists. A user has asked the publisher of the HaGeZi list whether they could be included in IPFire and under what terms. The response was that the list is available under the terms of the GNU General Public License v3, but that does not seem to be true. The list contains data from various sources. Many of them are licensed under incompatible licenses (CC BY-SA 4.0, MPL, Apache2, …) and unless there is a non-public agreement that this data may be redistributed, there is a huge legal issue here. We would expose our users to potential copyright infringement which we cannot do under any circumstances. Furthermore many lists are available under a non-commercial license which excludes them from being used in any kind of business. Plenty of IPFire systems are running in businesses, if not even the vast majority.
>>> 
>>> In short, these lists are completely unusable for us. Apart from HaGeZi, I consider OISD to have the same problem.
>>> 
>>> Enough about all the things that are bad. Let’s talk about the new, good things:
>>> 
>>> Many blacklists on the internet are an amalgamation of other lists. These lists vary in quality with some of them being not that good and without a clear focus and others being excellent data. Since we don’t have the man power to start from scratch, I felt that we can copy the concept that HaGeZi and OISD have started and simply create a new list that is based on other lists at the beginning to have a good starting point. That way, we have much better control over what is going on these lists and we can shape and mould them as we need them. Most importantly, we don’t create a single lists, but many lists that have a clear focus and allow users to choose what they want to block and what not.
>>> 
>>> So the current experimental stage that I am in has these lists:
>>> 
>>>   * Ads
>>>   * Dating
>>>   * DoH
>>>   * Gambling
>>>   * Malware
>>>   * Porn
>>>   * Social
>>>   * Violence
>>> 
>>> The categories have been determined by what source lists we have available with good data and are compatible with our chosen license CC BY-SA 4.0. This is the same license that we are using for the IPFire Location database, too.
>>> 
>>> The main use-cases for any kind of blocking are to comply with legal requirements in networks with children (i.e. schools) to remove any kind of pornographic content, sometimes block social media as well. Gambling and violence are commonly blocked, too. Even more common would be filtering advertising and any malicious content.
>>> 
>>> The latter is especially difficult because so many source lists throw phishing, spyware, malvertising, tracking and other things into the same bucket. Here this is currently all in the malware list which has therefore become quite large. I am not sure whether this will stay like this in the future or if we will have to make some adjustments, but that is exactly why this is now entering some larger testing.
>>> 
>>> What has been built so far? In order to put these lists together properly, track any data about where it is coming from, I have built a tool in Python available here:
>>> 
>>> https://git.ipfire.org/?p=dnsbl.git;a=summary
>>> 
>>> This tool will automatically update all lists once an hour if there have been any changes and export them in various formats. The exported lists are available for download here:
>>> 
>>> https://dnsbl.ipfire.org/lists/
>> The download using dnsbl.ipfire.org/lists/squidguard.tar.gz as the custom url works fine.
>> 
>> However you need to remember not to put the https:// at the front of the url otherwise the WUI page completes without any error messages but leaves an error message in the system logs saying
>> 
>> URL filter blacklist - ERROR: Not a valid URL filter blacklist
>> 
>> I found this out the hard way.
>> 
>> The other thing I noticed is that if you already have the Toulouse University list downloaded and you then change to the ipfire custom url then all the existing Toulouse blocklists stay in the directory on IPFire and so you end up with a huge number of category tick boxes, most of which are the old Toulouse ones, which are still available to select and it is not clear which ones are from Toulouse and which ones from IPFire.
>> 
>> I think if the blocklist URL source is changed or a custom url is provided the first step should be to remove the old ones already existing.
>> That might be a problem because users can also create their own blocklists and I believe those go into the same directory.
>> 
>> Without clearing out the old blocklists you end up with a huge number of checkboxes for lists but it is not clear what happens if there is a category that has the same name for the Toulouse list and the IPFire list such as gambling. I will have a look at that and see what happens.
>> 
>> Not sure what the best approach to this is.
> 
> 
> To find the older files I used this:
>    find /var/ipfire/urlfilter/blacklists -mtime +365 -type f -ls
> 
> To delete the older files and folders I did this:
>    find /var/ipfire/urlfilter/blacklists -mtime +365 -type f -delete -o -type d -empty -delete
> 
> 
>> 
>> 
>> Manually deleting all contents of the urlfilter/blacklists/ directory and then selecting the IPFire blocklist url for the custom url I end up with only the 8 categories from the IPFire list.
>> 
>> I have tested some gambling sites from the IPFire list and the block worked on some. On others the site no longer exists so there is nothing to block or has been changed to an https site and in that case it went straight through. Also if I chose the http version of the link, it was automatically changed to https and went through without being blocked.
>> 
>> 
>>> If you download and open any of the files, you will see a large header that includes copyright information and lists all sources that have been used to create the individual lists. This way we ensure maximum transparency, comply with the terms of the individual licenses of the source lists and give credit to the people who help us to put together the most perfect list for our users.
>>> 
>>> I would like this to become a project that is not only being used in IPFire. We can and will be compatible with other solutions like AdGuard, PiHole so that people can use our lists if they would like to even though they are not using IPFire. Hopefully, these users will also feed back to us so that we can improve our lists over time and make them one of the best options out there.
>>> 
>>> All lists are available as a simple text file that lists the domains. Then there is a hosts file available as well as a DNS zone file and an RPZ file. Each list is individually available to be used in squidGuard and there is a larger tarball available with all lists that can be used in IPFire’s URL Filter.
>> 
> I tested the URL Filter with the change to the autoupdate.urls.  For me it only picked up one URL but I think my Web Proxy is configured wrong.
> 
> • Is the non-Transparent needed to utilize the IPFire DNSBL (a.k.a. IPFire DNS Blocklists)??
> 
> • Does Web Proxy Auto-Discovery Protocol (WPAD) need to be setup?
> 
> I ask because I disabled the Web Proxy once Shalla Services stopped a few years ago.
> 
> I think the answer is yes to get HTTPS sites recognized.
> 
>> 
>>> I am planning to add Suricata/Snort signatures whenever I have time to do so. Even though it is not a good idea to filter pornographic content this way, I suppose that catching malware and blocking DoH are good use-cases for an IPS. Time will tell…
>>> 
>>> As a start, we will make these lists available in IPFire’s URL Filter and collect some feedback about how we are doing. Afterwards, we can see where else we can take this project.
>>> 
>>> If you want to enable this on your system, simply add the URL to your autoupdate.urls file like here:
>>> 
>>> https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=commitdiff;h=bf675bb937faa7617474b3cc84435af3b1f7f45f
>> I also tested out adding the IPFire url to autoupdate.urls and that also worked fine for me.
>> 
>> Regards,
>> Adolf.
>>> This email is just a brain dump from me to this list. I would be happy to answer any questions about implementation details, etc. if people are interested. Right now, this email is long enough already…
>>> 
>>> All the best,
>>> -Michael
>> 
>> -- Sent from my laptop
>> 
>> 
>>