[PATCH 1/3] sources: Removal of ALIENVAULT and SPAMHAUS_EDROP from ipblocklist sources

[PATCH 1/3] sources: Removal of ALIENVAULT and SPAMHAUS_EDROP from ipblocklist sources

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 3440 bytes --]

- ALIENVAULT has not been updated since at least Nov 2022 but probably earlier. There is no
   date for the file to be downloaded but a forum user has log messages from Nov 2022 that
   indicate the file had not changed as therefore no download occurred.
- AT&T aquired AlienVault in August 2018. Somewhere between 2018 and 2022 the list stopped
   getting updated. AlienVault references on the AT&T website are now for a different
   product.
- Discussed in IPFire conf call of April 2024 and agreed to remove the ALIENVAULT
   blocklist.
- On Apr 10th the Spamhaus eDROP list was merged with the Spamhaus DROP list. The eDROP
   list is still available but is now empty. Trying to select the SPAMHAUS_EDROP list
   gives an error message that the blocklist was found to be empty.
- This patch removes both the ALIENVAULT and the SPAMHAUS_EDROP lists from the ipblocklist
   sources file.

Tested-by: Adolf Belka <adolf.belka(a)ipfire.org>
Signed-off-by: Adolf Belka <adolf.belka(a)ipfire.org>
---
 config/ipblocklist/sources | 12 ------------
 1 file changed, 12 deletions(-)

diff --git a/config/ipblocklist/sources b/config/ipblocklist/sources
index be0cf0229..0835c0f9c 100644
--- a/config/ipblocklist/sources
+++ b/config/ipblocklist/sources
@@ -55,12 +55,6 @@ our %sources = ( 'EMERGING_FWRULE' => { 'name'     => 'Emerging Threats Blocklis
                                     'parser'   => 'ip-or-net-list',
                                     'rate'     => '12h',
                                     'category' => 'reputation' },
-             'SPAMHAUS_EDROP'  => { 'name'     => "Spamhaus Extended Don't Route or Peer List",
-                                    'url'      => 'https://www.spamhaus.org/drop/edrop.txt',
-                                    'info'     => 'https://www.spamhaus.org/drop/',
-                                    'parser'   => 'ip-or-net-list',
-                                    'rate'     => '1h',
-                                    'category' => 'reputation' },
              'DSHIELD'         => { 'name'     => 'Dshield.org Recommended Block List',
                                     'url'      => 'https://www.dshield.org/block.txt',
                                     'info'     => 'https://dshield.org/',
@@ -106,12 +100,6 @@ our %sources = ( 'EMERGING_FWRULE' => { 'name'     => 'Emerging Threats Blocklis
                                     'parser'   => 'ip-or-net-list',,
                                     'rate'     => '1h',
                                     'category' => 'application' },
-             'ALIENVAULT'      => { 'name'     => 'AlienVault IP Reputation database',
-                                    'url'      => 'https://reputation.alienvault.com/reputation.generic',
-                                    'info'     => 'https://www.alienvault.com/resource-center/videos/what-is-ip-domain-reputation',
-                                    'parser'   => 'ip-or-net-list',
-                                    'rate'     => '1h',
-                                    'category' => 'reputation' },
              'BOGON'           => { 'name'     => 'Bogus address list (Martian)',
                                     'url'      => 'https://www.team-cymru.org/Services/Bogons/bogon-bn-agg.txt',
                                     'info'     => 'https://www.team-cymru.com/bogon-reference',
-- 
2.44.0

[PATCH 2/3] update.sh: Remove existing entries for ALIENVAULT & SPAMHAUS_EDROP

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 1443 bytes --]

- This removes any time entries in the modified file for either ALIENVAULT or
   SPAMHAUS_EDROP.
- This also removes any blocklists for either of these sources from the /var/lib/ipblocklist
   directory.
- This patch will ensure that any reference to either of these sources is removed from the
   ipblocklist files.

Tested-by: Adolf Belka <adolf.belka(a)ipfire.org>
Signed-off-by: Adolf Belka <adolf.belka(a)ipfire.org>
---
 config/rootfiles/core/186/update.sh | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/config/rootfiles/core/186/update.sh b/config/rootfiles/core/186/update.sh
index d79a35133..9a4bd7f2a 100644
--- a/config/rootfiles/core/186/update.sh
+++ b/config/rootfiles/core/186/update.sh
@@ -123,6 +123,17 @@ if [ -e /boot/pakfire-kernel-update ]; then
     /boot/pakfire-kernel-update ${KVER}
 fi
 
+# Remove any entry for ALIENVAULT or SPAMHAUS_EDROP from the ipblocklist modified file
+# and the associated ipblocklist files from the /var/lib/ipblocklist directory
+sed -i '/ALIENVAULT=/d' /var/ipfire/ipblocklist/modified
+sed -i '/SPAMHAUS_EDROP=/d' /var/ipfire/ipblocklist/modified
+if [ -e /var/lib/ipblocklist/ALIENVAULT.conf ]; then
+	rm /var/lib/ipblocklist/ALIENVAULT.conf
+fi
+if [ -e /var/lib/ipblocklist/SPAMHAUS_EDROP.conf ]; then
+	rm /var/lib/ipblocklist/SPAMHAUS_EDROP.conf
+fi
+
 # This update needs a reboot...
 touch /var/run/need_reboot
 
-- 
2.44.0

[PATCH 3/3] backup.pl: removes any references to ALIENVAULT & SPAMHAUSEDROP from restores

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 1590 bytes --]

- This patch ensures that if a restore is carried out from an earlier version that includes
   ALIENVAULT and/or SPAMHAUS_EDROP that the references will be removed.
- This is the same code as was put into the update.sh file with the previous patch of this
   set.

Tested-by: Adolf Belka <adolf.belka(a)ipfire.org>
Signed-off-by: Adolf Belka <adolf.belka(a)ipfire.org>
---
 config/backup/backup.pl | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/config/backup/backup.pl b/config/backup/backup.pl
index b43420740..d848084e9 100644
--- a/config/backup/backup.pl
+++ b/config/backup/backup.pl
@@ -193,7 +193,7 @@ restore_backup() {
 
 	# Update OpenVPN CRL
 	/etc/fcron.daily/openvpn-crl-updater
-	
+
 	# Update OpenVPN N2N Client Configs
 	## Add providers legacy default line to n2n client config files
 	# Check if ovpnconfig exists and is not empty
@@ -246,6 +246,17 @@ restore_backup() {
 			-signkey /etc/httpd/server.key \
 			-out /etc/httpd/server.crt &>/dev/null
 	fi
+
+	# Remove any entry for ALIENVAULT or SPAMHAUS_EDROP from the ipblocklist modified file
+	# and the associated ipblocklist files from the /var/lib/ipblocklist directory
+	sed -i '/ALIENVAULT=/d' /var/ipfire/ipblocklist/modified
+	sed -i '/SPAMHAUS_EDROP=/d' /var/ipfire/ipblocklist/modified
+	if [ -e /var/lib/ipblocklist/ALIENVAULT.conf ]; then
+		rm /var/lib/ipblocklist/ALIENVAULT.conf
+	fi
+	if [ -e /var/lib/ipblocklist/SPAMHAUS_EDROP.conf ]; then
+		rm /var/lib/ipblocklist/SPAMHAUS_EDROP.conf
+	fi
 	return 0
 }
 
-- 
2.44.0

Re: [PATCH 1/3] sources: Removal of ALIENVAULT and SPAMHAUS_EDROP from ipblocklist sources

[Rob Brewer]

[-- Attachment #1: Type: text/plain, Size: 4341 bytes --]

On Fri, 19 Apr 2024 15:39:39 +0200, Adolf Belka wrote:

> - ALIENVAULT has not been updated since at least Nov 2022 but probably
> earlier. There is no
>    date for the file to be downloaded but a forum user has log messages
>    from Nov 2022 that indicate the file had not changed as therefore no
>    download occurred.
> - AT&T aquired AlienVault in August 2018. Somewhere between 2018 and
> 2022 the list stopped
>    getting updated. AlienVault references on the AT&T website are now
>    for a different product.
> - Discussed in IPFire conf call of April 2024 and agreed to remove the
> ALIENVAULT
>    blocklist.
> - On Apr 10th the Spamhaus eDROP list was merged with the Spamhaus DROP
> list. The eDROP
>    list is still available but is now empty. Trying to select the
>    SPAMHAUS_EDROP list gives an error message that the blocklist was
>    found to be empty.
> - This patch removes both the ALIENVAULT and the SPAMHAUS_EDROP lists
> from the ipblocklist
>    sources file.
> 
> Tested-by: Adolf Belka <adolf.belka(a)ipfire.org>
> Signed-off-by: Adolf Belka <adolf.belka(a)ipfire.org>
> ---
>  config/ipblocklist/sources | 12 ------------
>  1 file changed, 12 deletions(-)
> 
> diff --git a/config/ipblocklist/sources b/config/ipblocklist/sources
> index be0cf0229..0835c0f9c 100644 --- a/config/ipblocklist/sources +++
> b/config/ipblocklist/sources @@ -55,12 +55,6 @@ our %sources = (
> 'EMERGING_FWRULE' => { 'name'     => 'Emerging Threats Blocklis
>                                      'parser'   => 'ip-or-net-list',
>                                      'rate'     => '12h',
>                                      'category' => 'reputation' },
> -             'SPAMHAUS_EDROP'  => { 'name'     => "Spamhaus Extended
> Don't Route or Peer List",
> -                                    'url'      =>
> 'https://www.spamhaus.org/drop/edrop.txt',
> -                                    'info'     =>
> 'https://www.spamhaus.org/drop/',
> -                                    'parser'   => 'ip-or-net-list',
> -                                    'rate'     => '1h',
> -                                    'category' => 'reputation' },
>               'DSHIELD'         => { 'name'     => 'Dshield.org
>               Recommended Block List',
>                                      'url'      =>
>                                      'https://www.dshield.org/
block.txt',
>                                      'info'     =>
>                                      'https://dshield.org/',
> @@ -106,12 +100,6 @@ our %sources = ( 'EMERGING_FWRULE' => { 'name'    
> => 'Emerging Threats Blocklis
>                                      'parser'   => 'ip-or-net-list',,
>                                      'rate'     => '1h',
>                                      'category' => 'application' },
> -             'ALIENVAULT'      => { 'name'     => 'AlienVault IP
> Reputation database',
> -                                    'url'      =>
> 'https://reputation.alienvault.com/reputation.generic',
> -                                    'info'     =>
> 'https://www.alienvault.com/resource-center/videos/what-is-ip-domain-
reputation',
> -                                    'parser'   => 'ip-or-net-list',
> -                                    'rate'     => '1h',
> -                                    'category' => 'reputation' },
>               'BOGON'           => { 'name'     => 'Bogus address list
>               (Martian)',
>                                      'url'      =>
>                                      'https://www.team-cymru.org/
Services/Bogons/bogon-bn-agg.txt',


It would appear that SPAMHAUS_EDROP has been merged into SPAMHAUS_DROP 
list.

"; This list has been merged into https://www.spamhaus.org/drop/drop.txt
; Spamhaus EDROP List 2024/04/19 - (c) 2024 The Spamhaus Project
; https://www.spamhaus.org/drop/edrop.txt
; Last-Modified: Fri, 19 Apr 2024 13:49:21 GMT
; Expires: Sat, 20 Apr 2024 13:49:21 GMT
; EOF

I think it would be better to change the URL in the sources list from:

https://www.spamhaus.org/drop/edrop.txt

to

https://www.spamhaus.org/drop/drop.txt


Rather than just remove the list from the sources file.


Rob Brewer


>                                      'info'     =>
>                                      'https://www.team-cymru.com/bogon-
reference',

Re: [PATCH 1/3] sources: Removal of ALIENVAULT and SPAMHAUS_EDROP from ipblocklist sources

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 5742 bytes --]

Hi Rob,

On 20/04/2024 10:24, Rob Brewer wrote:
> On Fri, 19 Apr 2024 15:39:39 +0200, Adolf Belka wrote:
>
>> - ALIENVAULT has not been updated since at least Nov 2022 but probably
>> earlier. There is no
>>     date for the file to be downloaded but a forum user has log messages
>>     from Nov 2022 that indicate the file had not changed as therefore no
>>     download occurred.
>> - AT&T aquired AlienVault in August 2018. Somewhere between 2018 and
>> 2022 the list stopped
>>     getting updated. AlienVault references on the AT&T website are now
>>     for a different product.
>> - Discussed in IPFire conf call of April 2024 and agreed to remove the
>> ALIENVAULT
>>     blocklist.
>> - On Apr 10th the Spamhaus eDROP list was merged with the Spamhaus DROP
>> list. The eDROP
>>     list is still available but is now empty. Trying to select the
>>     SPAMHAUS_EDROP list gives an error message that the blocklist was
>>     found to be empty.
>> - This patch removes both the ALIENVAULT and the SPAMHAUS_EDROP lists
>> from the ipblocklist
>>     sources file.
>>
>> Tested-by: Adolf Belka <adolf.belka(a)ipfire.org>
>> Signed-off-by: Adolf Belka <adolf.belka(a)ipfire.org>
>> ---
>>   config/ipblocklist/sources | 12 ------------
>>   1 file changed, 12 deletions(-)
>>
>> diff --git a/config/ipblocklist/sources b/config/ipblocklist/sources
>> index be0cf0229..0835c0f9c 100644 --- a/config/ipblocklist/sources +++
>> b/config/ipblocklist/sources @@ -55,12 +55,6 @@ our %sources = (
>> 'EMERGING_FWRULE' => { 'name'     => 'Emerging Threats Blocklis
>>                                       'parser'   => 'ip-or-net-list',
>>                                       'rate'     => '12h',
>>                                       'category' => 'reputation' },
>> -             'SPAMHAUS_EDROP'  => { 'name'     => "Spamhaus Extended
>> Don't Route or Peer List",
>> -                                    'url'      =>
>> 'https://www.spamhaus.org/drop/edrop.txt',
>> -                                    'info'     =>
>> 'https://www.spamhaus.org/drop/',
>> -                                    'parser'   => 'ip-or-net-list',
>> -                                    'rate'     => '1h',
>> -                                    'category' => 'reputation' },
>>                'DSHIELD'         => { 'name'     => 'Dshield.org
>>                Recommended Block List',
>>                                       'url'      =>
>>                                       'https://www.dshield.org/
> block.txt',
>>                                       'info'     =>
>>                                       'https://dshield.org/',
>> @@ -106,12 +100,6 @@ our %sources = ( 'EMERGING_FWRULE' => { 'name'
>> => 'Emerging Threats Blocklis
>>                                       'parser'   => 'ip-or-net-list',,
>>                                       'rate'     => '1h',
>>                                       'category' => 'application' },
>> -             'ALIENVAULT'      => { 'name'     => 'AlienVault IP
>> Reputation database',
>> -                                    'url'      =>
>> 'https://reputation.alienvault.com/reputation.generic',
>> -                                    'info'     =>
>> 'https://www.alienvault.com/resource-center/videos/what-is-ip-domain-
> reputation',
>> -                                    'parser'   => 'ip-or-net-list',
>> -                                    'rate'     => '1h',
>> -                                    'category' => 'reputation' },
>>                'BOGON'           => { 'name'     => 'Bogus address list
>>                (Martian)',
>>                                       'url'      =>
>>                                       'https://www.team-cymru.org/
> Services/Bogons/bogon-bn-agg.txt',
>
>
> It would appear that SPAMHAUS_EDROP has been merged into SPAMHAUS_DROP
> list.
That is correct. That is what I put in the commit message.

Spamhaus have the following page about the change.

https://www.spamhaus.org/resource-hub/network-security/spamhaus-drop-and-edrop-to-become-a-single-list/#what-are-the-spamhaus-drop-lists

> "; This list has been merged into https://www.spamhaus.org/drop/drop.txt
> ; Spamhaus EDROP List 2024/04/19 - (c) 2024 The Spamhaus Project
> ; https://www.spamhaus.org/drop/edrop.txt
> ; Last-Modified: Fri, 19 Apr 2024 13:49:21 GMT
> ; Expires: Sat, 20 Apr 2024 13:49:21 GMT
> ; EOF
>
> I think it would be better to change the URL in the sources list from:
>
> https://www.spamhaus.org/drop/edrop.txt
>
> to
>
> https://www.spamhaus.org/drop/drop.txt
>
>
> Rather than just remove the list from the sources file.
I don't really understand your suggestion here. The EDROP list has gone. 
The old URL is still there but with an empty file except for the message.

The Spamhaus Drop list is now the equivalent of what used to be the 
Spamhaus eDrop list.

Having two entries, one called DROP and one EDROP both pointing to the 
same list seems pointless to me and potentially confusing for users as 
they might think they get something different from the two and if they 
select both they will get two sets of exactly the same IP's.

What I can do is to make a modification to the script I added to the 
update.sh file to check if SPAMHAUS_EDROP=on is set in the settings file 
and then add
SPAMHAUS_DROP=on to the settings file if it is not set, before removing 
the references to SPAMHAUS_EDROP.

Regards,

Adolf.

>
> Rob Brewer
>
>
>>                                       'info'     =>
>>                                       'https://www.team-cymru.com/bogon-
> reference',
>

-- 
Sent from my laptop

Re: [PATCH 1/3] sources: Removal of ALIENVAULT and SPAMHAUS_EDROP from ipblocklist sources

[Rob Brewer]

[-- Attachment #1: Type: text/plain, Size: 1273 bytes --]

On Sat, 20 Apr 2024 12:18:10 +0200, Adolf Belka wrote:


> I don't really understand your suggestion here. The EDROP list has gone.
> The old URL is still there but with an empty file except for the
> message.
> 
> The Spamhaus Drop list is now the equivalent of what used to be the
> Spamhaus eDrop list.
> 
> Having two entries, one called DROP and one EDROP both pointing to the
> same list seems pointless to me and potentially confusing for users as
> they might think they get something different from the two and if they
> select both they will get two sets of exactly the same IP's.
> 
> What I can do is to make a modification to the script I added to the
> update.sh file to check if SPAMHAUS_EDROP=on is set in the settings file
> and then add SPAMHAUS_DROP=on to the settings file if it is not set,
> before removing the references to SPAMHAUS_EDROP.
> 
> Regards,
> 
> Adolf

You are quite right! I misunderstood your patch because I am still on CU 
182 which doesn't have the SPAMHAUS_DROP updated sources list and was 
thinking that this was a removal  of the SPAMHAUS lists altogether. 

I see from my CU 184 system that both SPAMHAUS lists are present and 
therefore removal of SPAMHAUS_EDROP makes a lot of sense.

Sorry for the confusion.

Regards

Rob