Re: ipblacklist V2

Re: ipblacklist V2

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 5603 bytes --]

Hello Stefan,

Thank you for diving into this.

As you will all have seen, Stefan and Peter have recently migrated things away from xt_geoip and used ipset which is also being used for the blocklists. It would be great to combine all the code so that we have to maintain as little as possible.

On that note, something that I didn’t bring up yet, but which should probably be raised is that “blacklist” is probably not a good name any more in 2022. We use that term in many other places and didn’t change it yet, however, I would like to lead with a good example and add new code only under the more current names. I don’t have the intention to change the existing places myself, but I would be happy to accept a patch just so that we are consistent. I also believe that “blocklist” and “allowlist” describe the whole functionality much better than black and white.

I am not sure whether it is a good idea to rename everything rather sooner or at the end of the development process. Frankly it doesn’t matter, but it would be cool if the patchset put forward for merging into next would have this change.

On top of that I have a number of random things we would need to consider:

https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=config/kernel/kernel.config.x86_64-ipfire;hb=6e2c8f48182c169edb177526d7f639b0631d57cc#l1272

> CONFIG_IP_SET_MAX=256

This is the maximum number of ipsets the kernel is able to load at a time. Are we in any danger that we might hit this limit? If some (and yes, they shouldn’t, but the bloody do) enables all countries on the location filter that brings them already to 260. I would say that we should potentially raise this to 1024 or 2048 depending on how many people might make a lot of use of the AS block feature when that arrives.

I also stumbled over the hash size parameter. Stefan reported that the new DROP HOSTILE feature is slowing down his network, which is quite likely happening because of us hardcoding the hash size to 1024 (which is also the default for ipset).

Investigating any further, and testing a little bit around with memory overhead, I rewrite this in libloc:

  https://git.ipfire.org/?p=location/libloc.git;a=commitdiff;h=47de14b01a0fae69994131e4fb9239738570ab38

We are now counting the networks that we export (which is a little bit complicated, but doesn’t matter in this context) and then calculate the correct hash size with a factor. Choosing that factor is now the hard part and I have decided to start with 0.75. That means that we will definitely waste some memory, but we will hopefully not have to perform any linked list search whatsoever. Maybe this will need some adjustment, but it looks good to me. See the comment message for more details.

In IP blocklist, we have a factor of one which we potentially might want to look at as well, since those lists will be hit a lot. Has anyone conducted any benchmarks already?

-Michael

> On 28 Feb 2022, at 19:32, Stefan Schantl <stefan.schantl(a)ipfire.org> wrote:
> 
> Hello Rob, Hello Tim, Hello *,
> 
> as anounced on this list, I'm currently working on getting the
> ipblacklist feature as a core component into IPFire.
> 
> I already had a look on the code, which looks nice and very clean to
> me. As I'm currently also working on getting all ipset related set
> stuff and rule creation under one hood, this perfectly fits to this.
> 
> So my idea to put the ipblacklist feature over the line, was to split
> some parts of the ipblacklist "main script" (especially the ipset and
> iptables related stuff) into the perl-based script which is responsible
> for iptables rule creation.
> 
> In this case some other parts of the script (which where necessary in
> the past, because ipblacklist initial has been designed as an addon)
> also can be stipped.
> 
> Affected parts for example would be the "start", "stop", "enable" and
> "disable" code, which is not longer required and therefore safely can
> be dropped.
> 
> In the very end the main task for the script would be to download,
> update, convert and store the blacklists into an ipset compatible
> format.
> 
> Apart from this, I currently do not see any bigger changes for the WUI
> related stuff.
> 
> @Tim: I hope these changes are okay for you.
> 
> Getting started, I noticed, that there currently are two git
> repositories available, which contain the source for ipblacklist.
> 
> There is the origin one from Tim and a slightly modified (fixed) v3
> version from Rob. I' currently trying to determine, which one would be
> the best to start from - are there any deeper changes/differences
> between them?
> 
> Please feel free to ask any kind of questions or share you opinion. As
> usual, I'll share any progress here.
> 
> Best regards,
> 
> -Stefan 
>> Hi.
>> 
>> I have been looking at Tim FitzGeorge's code for ipblacklist v2 on 
>> https://patchwork.ipfire.org/project/ipfire/list/?series=1215 to see
>> if I 
>> can help progress its incorporation into IPFire.
>> After I extracted the programs from Patchwork I have been able to
>> build them 
>> into my firewall where they are running very successfully.
>> The code on the server seems to be in good shape and apart from a few
>> small 
>> patches and additions of a few missing scripts I think it could be 
>> successfully introduced into the IPFire code base.
>> I am more than happy to help in seeing this process carried out but
>> need to 
>> know if this is acceptable to yourselves.
>> 
>> Regards
>> Rob Brewer
> 
> 

Re: ipblacklist V2

[Rob Brewer]

[-- Attachment #1: Type: text/plain, Size: 3240 bytes --]

Hi Stefan

On Monday 28 February 2022 19:32 Stefan Schantl wrote:

> Hello Rob, Hello Tim, Hello *,
> 
> as anounced on this list, I'm currently working on getting the
> ipblacklist feature as a core component into IPFire.
> 
> I already had a look on the code, which looks nice and very clean to
> me. As I'm currently also working on getting all ipset related set
> stuff and rule creation under one hood, this perfectly fits to this.
> 
I have been in contact with Michael and has kept me up to date with your 
progress. As I have found out during a recent DOS attack @Tim's ipblacklist is 
very powerful tool in combating unwanted internet traffic. I had been using 
@Tim's original ipfblocklist addon which had limitations when trying to use 
larger blocklists, which @Tim had fixed in his later ipblacklist version. 

Using @Tims repositories I was able to recover the code and install ipblacklist 
on my current firewall (core 161) and confirm that the code was fully 
functional.

Looking back at the history of ipblacklist the first version contained a dynamic 
blocklist feature which was dropped in favour of the one now available (V2) but 
was a patched version of the original V1 code. 

My V3 version is a rework of @Tim's V2 code with the introduction of some of the 
missing unchanged patches from the V1 code plus updated patches to track the 
core changes from when it was last worked on about 2 years ago.

> So my idea to put the ipblacklist feature over the line, was to split
> some parts of the ipblacklist "main script" (especially the ipset and
> iptables related stuff) into the perl-based script which is responsible
> for iptables rule creation.
> 
That seems to be a sensible approach.

> In this case some other parts of the script (which where necessary in
> the past, because ipblacklist initial has been designed as an addon)
> also can be stipped.
> 
> Affected parts for example would be the "start", "stop", "enable" and
> "disable" code, which is not longer required and therefore safely can
> be dropped.
> 

I think that is correct. There are several processes that are started and 
stopped in the firewall script but need to be run but in a controlled order. 


> In the very end the main task for the script would be to download,
> update, convert and store the blacklists into an ipset compatible
> format.
> 
> Apart from this, I currently do not see any bigger changes for the WUI
> related stuff.
> 
> @Tim: I hope these changes are okay for you.
> 

> Getting started, I noticed, that there currently are two git
> repositories available, which contain the source for ipblacklist.
> 
> There is the origin one from Tim and a slightly modified (fixed) v3
> version from Rob. I' currently trying to determine, which one would be
> the best to start from - are there any deeper changes/differences
> between them?
> 
I have integrated my V3 patches into core 163 and have a build running on an 
apu2 new install. I haven't found any problems with this install.


> Please feel free to ask any kind of questions or share you opinion. As
> usual, I'll share any progress here.
> 
> Best regards,
> 
> -Stefan


Rob

Re: ipblacklist V2

[Stefan Schantl]

[-- Attachment #1: Type: text/plain, Size: 3626 bytes --]

Hello *,

I've made some development progress, which I want to share here:

Most parts of the main backend script ("ipblacklist") from Tim and Rob
are ported into a new functions library (ipblocklist-functions.pl) and
into the main firewall script (rules.pl).

This process is almost finished and currently allows to create the
firewall rules, download the blocklists and to convert them into an
ipset compatible format.

Next step will be to import the frontend code (WUI) and adjust it to
use the backend code (functions) from the "ipblocklist-functions.pl".

At this time the blocklist feature should be in a use-able state again
and I'll go to create an automatic update script and to import all the
logging pages stuff etc.

The development progress and single commits can be found here:

https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=shortlog;h=refs/heads/ipblocklist

As usual please feel free to ask any questions or to share your opinion
here.

I wish you a nice day,

-Stefan
> Hello Rob, Hello Tim, Hello *,
> 
> as anounced on this list, I'm currently working on getting the
> ipblacklist feature as a core component into IPFire.
> 
> I already had a look on the code, which looks nice and very clean to
> me. As I'm currently also working on getting all ipset related set
> stuff and rule creation under one hood, this perfectly fits to this.
> 
> So my idea to put the ipblacklist feature over the line, was to split
> some parts of the ipblacklist "main script" (especially the ipset and
> iptables related stuff) into the perl-based script which is
> responsible
> for iptables rule creation.
> 
> In this case some other parts of the script (which where necessary in
> the past, because ipblacklist initial has been designed as an addon)
> also can be stipped.
> 
> Affected parts for example would be the "start", "stop", "enable" and
> "disable" code, which is not longer required and therefore safely can
> be dropped.
> 
> In the very end the main task for the script would be to download,
> update, convert and store the blacklists into an ipset compatible
> format.
> 
> Apart from this, I currently do not see any bigger changes for the
> WUI
> related stuff.
> 
> @Tim: I hope these changes are okay for you.
> 
> Getting started, I noticed, that there currently are two git
> repositories available, which contain the source for ipblacklist.
> 
> There is the origin one from Tim and a slightly modified (fixed) v3
> version from Rob. I' currently trying to determine, which one would
> be
> the best to start from - are there any deeper changes/differences
> between them?
> 
> Please feel free to ask any kind of questions or share you opinion.
> As
> usual, I'll share any progress here.
> 
> Best regards,
> 
> -Stefan 
> > Hi.
> > 
> > I have been looking at Tim FitzGeorge's code for ipblacklist v2 on 
> > https://patchwork.ipfire.org/project/ipfire/list/?series=1215 to
> > see
> > if I 
> > can help progress its incorporation into IPFire.
> > After I extracted the programs from Patchwork I have been able to
> > build them 
> > into my firewall where they are running very successfully.
> > The code on the server seems to be in good shape and apart from a
> > few
> > small 
> > patches and additions of a few missing scripts I think it could be 
> > successfully introduced into the IPFire code base.
> > I am more than happy to help in seeing this process carried out but
> > need to 
> > know if this is acceptable to yourselves.
> > 
> > Regards
> > Rob Brewer
> 
> 

Re: ipblacklist V2

[Rob Brewer]

[-- Attachment #1: Type: text/plain, Size: 1613 bytes --]

Hi Stefan

On Saturday 05 March 2022 18:52 Stefan Schantl wrote:

> Hello *,
> 
> I've made some development progress, which I want to share here:
> 
> Most parts of the main backend script ("ipblacklist") from Tim and Rob
> are ported into a new functions library (ipblocklist-functions.pl) and
> into the main firewall script (rules.pl).
> 
Good
> This process is almost finished and currently allows to create the
> firewall rules, download the blocklists and to convert them into an
> ipset compatible format.
> 
> Next step will be to import the frontend code (WUI) and adjust it to
> use the backend code (functions) from the "ipblocklist-functions.pl".
> 
> At this time the blocklist feature should be in a use-able state again
> and I'll go to create an automatic update script and to import all the
> logging pages stuff etc.
> 
> The development progress and single commits can be found here:
> 
> 
https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=shortlog;h=refs/heads/ipblocklist
> 
> As usual please feel free to ask any questions or to share your opinion
> here.
> 
> I wish you a nice day,
> 
> -Stefan
Great progress. I did find a bug I introduced when I modified the ipblacklist V2 
perl script to add a space after the log-prefix BLKLST entry to make the logs 
compatible with other log-prefixes. 
This affected showrequestfromblacklist.dat and the modified version 
'modified regex for V3 log-prefix added space' should be used.

https://git.ipfire.org/?p=people/helix/ipfire-2.x.git;a=commit;h=2ccc47f1944fcf1f8eec8ad132fa606081b21578

Rob

Re: ipblacklist V2

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1919 bytes --]

Hello Rob,

> On 5 Mar 2022, at 21:46, Rob Brewer <ipfire-devel(a)grantura.co.uk> wrote:
> 
> Hi Stefan
> 
> On Saturday 05 March 2022 18:52 Stefan Schantl wrote:
> 
>> Hello *,
>> 
>> I've made some development progress, which I want to share here:
>> 
>> Most parts of the main backend script ("ipblacklist") from Tim and Rob
>> are ported into a new functions library (ipblocklist-functions.pl) and
>> into the main firewall script (rules.pl).
>> 
> Good
>> This process is almost finished and currently allows to create the
>> firewall rules, download the blocklists and to convert them into an
>> ipset compatible format.
>> 
>> Next step will be to import the frontend code (WUI) and adjust it to
>> use the backend code (functions) from the "ipblocklist-functions.pl".
>> 
>> At this time the blocklist feature should be in a use-able state again
>> and I'll go to create an automatic update script and to import all the
>> logging pages stuff etc.
>> 
>> The development progress and single commits can be found here:
>> 
>> 
> https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=shortlog;h=refs/heads/ipblocklist
>> 
>> As usual please feel free to ask any questions or to share your opinion
>> here.
>> 
>> I wish you a nice day,
>> 
>> -Stefan
> Great progress. I did find a bug I introduced when I modified the ipblacklist V2 
> perl script to add a space after the log-prefix BLKLST entry to make the logs 
> compatible with other log-prefixes. 
> This affected showrequestfromblacklist.dat and the modified version 
> 'modified regex for V3 log-prefix added space' should be used.
> 
> https://git.ipfire.org/?p=people/helix/ipfire-2.x.git;a=commit;h=2ccc47f1944fcf1f8eec8ad132fa606081b21578

Good catch, but wouldn’t it be helpful to add the space to all log prefixes so that it is always easily readable for humans, too?

-Michael

> 
> Rob

Re: ipblacklist V2

[Rob Brewer]

[-- Attachment #1: Type: text/plain, Size: 2097 bytes --]

On Monday 07 March 2022 20:39 Michael Tremer wrote:

> Hello Rob,
> 
>> On 5 Mar 2022, at 21:46, Rob Brewer <ipfire-devel(a)grantura.co.uk> wrote:
>> 
>> Hi Stefan
>> 
>> On Saturday 05 March 2022 18:52 Stefan Schantl wrote:
>> 
>>> Hello *,
>>> 
>>> I've made some development progress, which I want to share here:
>>> 
>>> Most parts of the main backend script ("ipblacklist") from Tim and Rob
>>> are ported into a new functions library (ipblocklist-functions.pl) and
>>> into the main firewall script (rules.pl).
>>> 
>> Good
>>> This process is almost finished and currently allows to create the
>>> firewall rules, download the blocklists and to convert them into an
>>> ipset compatible format.
>>> 
>>> Next step will be to import the frontend code (WUI) and adjust it to
>>> use the backend code (functions) from the "ipblocklist-functions.pl".
>>> 
>>> At this time the blocklist feature should be in a use-able state again
>>> and I'll go to create an automatic update script and to import all the
>>> logging pages stuff etc.
>>> 
>>> The development progress and single commits can be found here:
>>> 
>>> 
>> https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=shortlog;h=refs/heads/ipblocklist
>>> 
>>> As usual please feel free to ask any questions or to share your opinion
>>> here.
>>> 
>>> I wish you a nice day,
>>> 
>>> -Stefan
>> Great progress. I did find a bug I introduced when I modified the ipblacklist
>> V2 perl script to add a space after the log-prefix BLKLST entry to make the
>> logs compatible with other log-prefixes.
>> This affected showrequestfromblacklist.dat and the modified version
>> 'modified regex for V3 log-prefix added space' should be used.
>> 
>> https://git.ipfire.org/?p=people/helix/ipfire-2.x.git;a=commit;h=2ccc47f1944fcf1f8eec8ad132fa606081b21578
> 
> Good catch, but wouldn’t it be helpful to add the space to all log prefixes so
> that it is always easily readable for humans, too?
> 
> -Michael
> 

I think ipblocklist was the only logfile entry to have this problem.

Rob

Re: ipblacklist V2

[Rob Brewer]

[-- Attachment #1: Type: text/plain, Size: 2805 bytes --]

On Monday 07 March 2022 22:54 Rob Brewer wrote:

> On Monday 07 March 2022 20:39 Michael Tremer wrote:
> 
>> Hello Rob,
>> 
>>> On 5 Mar 2022, at 21:46, Rob Brewer <ipfire-devel(a)grantura.co.uk> wrote:
>>> 
>>> Hi Stefan
>>> 
>>> On Saturday 05 March 2022 18:52 Stefan Schantl wrote:
>>> 
>>>> Hello *,
>>>> 
>>>> I've made some development progress, which I want to share here:
>>>> 
>>>> Most parts of the main backend script ("ipblacklist") from Tim and Rob
>>>> are ported into a new functions library (ipblocklist-functions.pl) and
>>>> into the main firewall script (rules.pl).
>>>> 
>>> Good
>>>> This process is almost finished and currently allows to create the
>>>> firewall rules, download the blocklists and to convert them into an
>>>> ipset compatible format.
>>>> 
>>>> Next step will be to import the frontend code (WUI) and adjust it to
>>>> use the backend code (functions) from the "ipblocklist-functions.pl".
>>>> 
>>>> At this time the blocklist feature should be in a use-able state again
>>>> and I'll go to create an automatic update script and to import all the
>>>> logging pages stuff etc.
>>>> 
>>>> The development progress and single commits can be found here:
>>>> 
>>>> 
>>> https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=shortlog;h=refs/heads/ipblocklist
>>>> 
>>>> As usual please feel free to ask any questions or to share your opinion
>>>> here.
>>>> 
>>>> I wish you a nice day,
>>>> 
>>>> -Stefan
>>> Great progress. I did find a bug I introduced when I modified the
>>> ipblacklist V2 perl script to add a space after the log-prefix BLKLST entry
>>> to make the logs compatible with other log-prefixes.
>>> This affected showrequestfromblacklist.dat and the modified version
>>> 'modified regex for V3 log-prefix added space' should be used.
>>> 
>>> https://git.ipfire.org/?p=people/helix/ipfire-2.x.git;a=commit;h=2ccc47f1944fcf1f8eec8ad132fa606081b21578
>> 
>> Good catch, but wouldn’t it be helpful to add the space to all log prefixes
>> so that it is always easily readable for humans, too?
>> 
>> -Michael
>> 
> 
> I think ipblocklist was the only logfile entry to have this problem.
> 
> Rob

I think there is a missing space in  rules.pl before tha last "\"
I think line 755 should be:

run("$IPTABLES -A ${blocklist}_DROP -j LOG -m limit --limit 10/second --log-prefix \"BLKLST_$blocklist \"");


and in my ipblocklists.dat I changed your line 298 to be compatible with core 163 as I don't think you can change the 'theme' now
and produced an error.

from:
 &General::readhash("/srv/web/ipfire/html/themes/".$mainsettings{'THEME'}."/include/colors.txt", \%color);

to: 
 &General::readhash("/srv/web/ipfire/html/themes/ipfire/include/colors.txt", \%color);


Rob

Re: ipblacklist V2

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 3169 bytes --]

Hello Rob,

> On 8 Mar 2022, at 10:59, Rob Brewer <ipfire-devel(a)grantura.co.uk> wrote:
> 
> On Monday 07 March 2022 22:54 Rob Brewer wrote:
> 
>> On Monday 07 March 2022 20:39 Michael Tremer wrote:
>> 
>>> Hello Rob,
>>> 
>>>> On 5 Mar 2022, at 21:46, Rob Brewer <ipfire-devel(a)grantura.co.uk> wrote:
>>>> 
>>>> Hi Stefan
>>>> 
>>>> On Saturday 05 March 2022 18:52 Stefan Schantl wrote:
>>>> 
>>>>> Hello *,
>>>>> 
>>>>> I've made some development progress, which I want to share here:
>>>>> 
>>>>> Most parts of the main backend script ("ipblacklist") from Tim and Rob
>>>>> are ported into a new functions library (ipblocklist-functions.pl) and
>>>>> into the main firewall script (rules.pl).
>>>>> 
>>>> Good
>>>>> This process is almost finished and currently allows to create the
>>>>> firewall rules, download the blocklists and to convert them into an
>>>>> ipset compatible format.
>>>>> 
>>>>> Next step will be to import the frontend code (WUI) and adjust it to
>>>>> use the backend code (functions) from the "ipblocklist-functions.pl".
>>>>> 
>>>>> At this time the blocklist feature should be in a use-able state again
>>>>> and I'll go to create an automatic update script and to import all the
>>>>> logging pages stuff etc.
>>>>> 
>>>>> The development progress and single commits can be found here:
>>>>> 
>>>>> 
>>>> https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=shortlog;h=refs/heads/ipblocklist
>>>>> 
>>>>> As usual please feel free to ask any questions or to share your opinion
>>>>> here.
>>>>> 
>>>>> I wish you a nice day,
>>>>> 
>>>>> -Stefan
>>>> Great progress. I did find a bug I introduced when I modified the
>>>> ipblacklist V2 perl script to add a space after the log-prefix BLKLST entry
>>>> to make the logs compatible with other log-prefixes.
>>>> This affected showrequestfromblacklist.dat and the modified version
>>>> 'modified regex for V3 log-prefix added space' should be used.
>>>> 
>>>> https://git.ipfire.org/?p=people/helix/ipfire-2.x.git;a=commit;h=2ccc47f1944fcf1f8eec8ad132fa606081b21578
>>> 
>>> Good catch, but wouldn’t it be helpful to add the space to all log prefixes
>>> so that it is always easily readable for humans, too?
>>> 
>>> -Michael
>>> 
>> 
>> I think ipblocklist was the only logfile entry to have this problem.
>> 
>> Rob
> 
> I think there is a missing space in  rules.pl before tha last "\"
> I think line 755 should be:
> 
> run("$IPTABLES -A ${blocklist}_DROP -j LOG -m limit --limit 10/second --log-prefix \"BLKLST_$blocklist \"");

Yes, that is the space character I am talking about.

> and in my ipblocklists.dat I changed your line 298 to be compatible with core 163 as I don't think you can change the 'theme' now
> and produced an error.
> 
> from:
> &General::readhash("/srv/web/ipfire/html/themes/".$mainsettings{'THEME'}."/include/colors.txt", \%color);
> 
> to: 
> &General::readhash("/srv/web/ipfire/html/themes/ipfire/include/colors.txt", \%color);

No, this has indeed been dropped and needs to be updated in the CGI. Let’s leave that to Stefan :)

-Michael

> 
> 
> Rob
> 

Re: ipblacklist V2

[Stefan Schantl]

[-- Attachment #1: Type: text/plain, Size: 3983 bytes --]

Hello Rob and list,

sorry for the long delay. Finally all major problems around the
latest IDS improvements and the release around them has passed, so
let's get back to the ipblocklist feature.

> On Monday 07 March 2022 22:54 Rob Brewer wrote:
> 
> > On Monday 07 March 2022 20:39 Michael Tremer wrote:
> > 
> > > Hello Rob,
> > > 
> > > > On 5 Mar 2022, at 21:46, Rob Brewer < 
> > > > ipfire-devel(a)grantura.co.uk> wrote:
> > > > 
> > > > Hi Stefan
> > > > 
> > > > On Saturday 05 March 2022 18:52 Stefan Schantl wrote:
> > > > 
> > > > > Hello *,
> > > > > 
> > > > > I've made some development progress, which I want to share
> > > > > here:
> > > > > 
> > > > > Most parts of the main backend script ("ipblacklist") from
> > > > > Tim and Rob
> > > > > are ported into a new functions library (ipblocklist-
> > > > > functions.pl) and
> > > > > into the main firewall script (rules.pl).
> > > > > 
> > > > Good
> > > > > This process is almost finished and currently allows to
> > > > > create the
> > > > > firewall rules, download the blocklists and to convert them
> > > > > into an
> > > > > ipset compatible format.
> > > > > 
> > > > > Next step will be to import the frontend code (WUI) and
> > > > > adjust it to
> > > > > use the backend code (functions) from the "ipblocklist-
> > > > > functions.pl".
> > > > > 
> > > > > At this time the blocklist feature should be in a use-able
> > > > > state again
> > > > > and I'll go to create an automatic update script and to
> > > > > import all the
> > > > > logging pages stuff etc.
> > > > > 
> > > > > The development progress and single commits can be found
> > > > > here:
> > > > > 
> > > > > 
> > > >  
> > > > https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=shortlog;h=refs/heads/ipblocklist
> > > > > 
> > > > > As usual please feel free to ask any questions or to share
> > > > > your opinion
> > > > > here.
> > > > > 
> > > > > I wish you a nice day,
> > > > > 
> > > > > -Stefan
> > > > Great progress. I did find a bug I introduced when I modified
> > > > the
> > > > ipblacklist V2 perl script to add a space after the log-prefix
> > > > BLKLST entry
> > > > to make the logs compatible with other log-prefixes.
> > > > This affected showrequestfromblacklist.dat and the modified
> > > > version
> > > > 'modified regex for V3 log-prefix added space' should be used.
> > > > 
> > > >  
> > > > https://git.ipfire.org/?p=people/helix/ipfire-2.x.git;a=commit;h=2ccc47f1944fcf1f8eec8ad132fa606081b21578

Thanks for the fix, I've picked it up and added it to the current
development branch.

https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=80a456334fb24059e1d3585b26318b1e6124aa6f

> > > 
> > > Good catch, but wouldn’t it be helpful to add the space to all
> > > log prefixes
> > > so that it is always easily readable for humans, too?
> > > 
> > > -Michael
> > > 
> > 
> > I think ipblocklist was the only logfile entry to have this
> > problem.
> > 
> > Rob
> 
> I think there is a missing space in  rules.pl before tha last "\"
> I think line 755 should be:
> 
> run("$IPTABLES -A ${blocklist}_DROP -j LOG -m limit --limit 10/second
> --log-prefix \"BLKLST_$blocklist \"");

Thanks for pointing this out - see:

https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=84642aadeb63eaa9c5da0e6b3236bc6731a63580

> 
> 
> and in my ipblocklists.dat I changed your line 298 to be compatible
> with core 163 as I don't think you can change the 'theme' now
> and produced an error.
> 
> from:
>  &General::readhash("/srv/web/ipfire/html/themes/".$mainsettings{'THE
> ME'}."/include/colors.txt", \%color);
> 
> to: 
>  &General::readhash("/srv/web/ipfire/html/themes/ipfire/include/color
> s.txt", \%color);


Fixed:

https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=acf4a99d23576556727cffabc0089306d47b2b61

> 
> 
> Rob
> 

Re: ipblacklist V2

[Rob Brewer]

[-- Attachment #1: Type: text/plain, Size: 4909 bytes --]

Hi Stefan,

Thank you for your update.

As you probably realize I have ben using  Tim's IPblacklist software modified 
with the below changes for the past few weeks without problems so when it is 
released into IPFire, I wouldn't expect there to be any problems. Do you have an 
idea of the release issue you are aiming for for the IPFire version?

I have now integrated my ported Banish addon into IPblacklist which allows the 
user to maintain a personnel GUI based  Banish list through the IPBlacklist 
interface which I find extremely useful. It updates considerably faster now it 
is using ipset rather than the original IPTables. I'll add the details to my git 
pages in the next few days.


Rob


On Sunday 03 April 2022 10:16 Stefan Schantl wrote:

> Hello Rob and list,
> 
> sorry for the long delay. Finally all major problems around the
> latest IDS improvements and the release around them has passed, so
> let's get back to the ipblocklist feature.
> 
>> On Monday 07 March 2022 22:54 Rob Brewer wrote:
>> 
>> > On Monday 07 March 2022 20:39 Michael Tremer wrote:
>> > 
>> > > Hello Rob,
>> > > 
>> > > > On 5 Mar 2022, at 21:46, Rob Brewer <
>> > > > ipfire-devel(a)grantura.co.uk> wrote:
>> > > > 
>> > > > Hi Stefan
>> > > > 
>> > > > On Saturday 05 March 2022 18:52 Stefan Schantl wrote:
>> > > > 
>> > > > > Hello *,
>> > > > > 
>> > > > > I've made some development progress, which I want to share
>> > > > > here:
>> > > > > 
>> > > > > Most parts of the main backend script ("ipblacklist") from
>> > > > > Tim and Rob
>> > > > > are ported into a new functions library (ipblocklist-
>> > > > > functions.pl) and
>> > > > > into the main firewall script (rules.pl).
>> > > > > 
>> > > > Good
>> > > > > This process is almost finished and currently allows to
>> > > > > create the
>> > > > > firewall rules, download the blocklists and to convert them
>> > > > > into an
>> > > > > ipset compatible format.
>> > > > > 
>> > > > > Next step will be to import the frontend code (WUI) and
>> > > > > adjust it to
>> > > > > use the backend code (functions) from the "ipblocklist-
>> > > > > functions.pl".
>> > > > > 
>> > > > > At this time the blocklist feature should be in a use-able
>> > > > > state again
>> > > > > and I'll go to create an automatic update script and to
>> > > > > import all the
>> > > > > logging pages stuff etc.
>> > > > > 
>> > > > > The development progress and single commits can be found
>> > > > > here:
>> > > > > 
>> > > > > 
>> > > >  
>> > > > 
https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=shortlog;h=refs/heads/ipblocklist
>> > > > > 
>> > > > > As usual please feel free to ask any questions or to share
>> > > > > your opinion
>> > > > > here.
>> > > > > 
>> > > > > I wish you a nice day,
>> > > > > 
>> > > > > -Stefan
>> > > > Great progress. I did find a bug I introduced when I modified
>> > > > the
>> > > > ipblacklist V2 perl script to add a space after the log-prefix
>> > > > BLKLST entry
>> > > > to make the logs compatible with other log-prefixes.
>> > > > This affected showrequestfromblacklist.dat and the modified
>> > > > version
>> > > > 'modified regex for V3 log-prefix added space' should be used.
>> > > > 
>> > > >  
>> > > > 
https://git.ipfire.org/?p=people/helix/ipfire-2.x.git;a=commit;h=2ccc47f1944fcf1f8eec8ad132fa606081b21578
> 
> Thanks for the fix, I've picked it up and added it to the current
> development branch.
> 
> 
https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=80a456334fb24059e1d3585b26318b1e6124aa6f
> 
>> > > 
>> > > Good catch, but wouldn’t it be helpful to add the space to all
>> > > log prefixes
>> > > so that it is always easily readable for humans, too?
>> > > 
>> > > -Michael
>> > > 
>> > 
>> > I think ipblocklist was the only logfile entry to have this
>> > problem.
>> > 
>> > Rob
>> 
>> I think there is a missing space in  rules.pl before tha last "\"
>> I think line 755 should be:
>> 
>> run("$IPTABLES -A ${blocklist}_DROP -j LOG -m limit --limit 10/second
>> --log-prefix \"BLKLST_$blocklist \"");
> 
> Thanks for pointing this out - see:
> 
> 
https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=84642aadeb63eaa9c5da0e6b3236bc6731a63580
> 
>> 
>> 
>> and in my ipblocklists.dat I changed your line 298 to be compatible
>> with core 163 as I don't think you can change the 'theme' now
>> and produced an error.
>> 
>> from:
>> &General::readhash("/srv/web/ipfire/html/themes/".$mainsettings{'THE
>> ME'}."/include/colors.txt", \%color);
>> 
>> to:
>> &General::readhash("/srv/web/ipfire/html/themes/ipfire/include/color
>> s.txt", \%color);
> 
> 
> Fixed:
> 
> 
https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=acf4a99d23576556727cffabc0089306d47b2b61
> 
>> 
>> 
>> Rob
>>

Re: ipblacklist V2

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 3459 bytes --]

Hello,

> On 13 Feb 2022, at 12:44, Rob Brewer <ipfire-devel(a)grantura.co.uk> wrote:
> 
> Hi Tim, Good to see your posing here.
> 
> On Saturday 12 February 2022 21:29 Tim FitzGeorge wrote:
> 
>> This sounds as if it does the same sort of thing as something I had in my first patchset.  I added an extra rule to the input policy chain that added the address to an ipset if the number of dropped packets exceeded a 
> threshold.  This runs completely within iptables/ipset.
>> 
>> iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist src -j SET --add-set $autoblacklist src --exist" );
>> iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist dst -j SET --add-set $autoblacklist dst --exist" );
>> iptables( "-I POLICYIN 1 -i $red_iface -m hashlimit --hashlimit-mode srcip --hashlimit-above $settings{BLOCK_THRESHOLD}/hour --hashlimit-name $autoblacklist -j SET --add-set $autoblacklist src" );
>> 
> 
> Aotoblacklist looks like a useful addition. Why did you drop t?

I asked for that. I believed that it would causing more trouble than it helps.

The default policy of the firewall is dropping traffic anyways. Adding an extra rule does not change behaviour of the firewall and I wasn’t sure if it would add some false sense of extra security.

Since the firewall has no idea who is hammering some web application trying out passwords I did not see the value if that makes sense.

It would be great to have some more insight into who the big offenders are, but I don’t believe that this is the right place.

> 
> 
>>>>> 
>>>> There are a couple of points we need to consider:
>>>> 
>>>> 1) IPBlacklist does not work very well if Tim's  ipfblocklist add-on is also 
>>>> installed. My view is that the add-on should be removed before IPBlacklist 
>>>> can be applied. Can the add-on be automatically removed on installaion and 
>>>> should we transfer the settings info from ipfbocklist to ipblacklist?
>>> 
>>> Yes, in theory we could remove any old files in the updater and install our own ones.
>>> 
> 
> There are a couple of errors on your uninstall-blocklist.sh script which leaves
> some files behind when it is run. I can send you a patch for this if it is of help.

Since this is going into the main distribution we won’t need this script any more, but you can make it work for testing if you like.

>>>> 2) I added a init script to my firewall which doesn't seem to be present on 
>>>> Tim's patches. I'm not sure if this is needed as it will be started by fcron 
>>>> or changes made in the WUI but won't be instantly available on re-boot. Do 
>>>> you have any thoughts on this?
>>> 
>> 
>> I don't think this is needed - the change to the firewall init script should call the ipblacklist script at the correct time.
> 
> I hadn't noticed the last few lines in your firewall init script which my init
> script duplicates. So I agree my addition isn't needed.
> 
> I have started producing the v3 patches requested by the devs, but apart from the couple 
> of changes needed to  ipblacklists.dat I think they will be almost identical to your
> v2 patches.

It isn’t bad when things are identical. Good code is good code and doesn’t need to be changed just for the sake of it.

Your Git repository is still empty, do you know how to push anything into it? I am mostly curious if I set it up okay that it will work :)

-Michael

> Rob 

Re: ipblacklist V2

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 7707 bytes --]

Hello Tim,

Welcome back :)

> On 12 Feb 2022, at 21:29, Tim FitzGeorge <ipfr(a)tfitzgeorge.me.uk> wrote:
> 
> Hello,
> 
> On 10/02/2022 16:48, Michael Tremer wrote:
>> Hello Rob,
>> 
>>> On 10 Feb 2022, at 15:12, Rob Brewer <ipfire-devel(a)grantura.co.uk> wrote:
>>> 
>>> Hi Michael
>>> 
>>> On Thursday 10 February 2022 09:41 Michael Tremer wrote:
>>> 
>>>> Hello Rob,
>>>> 
>>>> 
>>>> Good to hear that you are in touch. I would like to invite Tim to join the
>>>> conversation on here. I am sure he has a couple of thoughts to contribute
>>>> and I hope he can find the time.
>>>> 
>>>> 
>>>> I assume you are talking about this here?
>>>> 
>>>> 
>>> https://git.ipfire.org/?p=people/timf/ipfire-2.x.git;a=shortlog;h=refs/heads/ipblacklist
>>> 
>>> Yes Tim's link to his ipfire git pages which links your URL.
>> 
>> Okay, I just set up your very own repository over here:
>> 
>>  https://git.ipfire.org/?p=people/helix/ipfire-2.x.git;a=summary
>> 
>> I will send you a private email with your account stuff.
>> 
>>>> 
>>>> That would have been one of my first questions having looked at my emails
>>>> again: Get the code into some Git repository.
>>>> 
>>>> This is a large patchset and it is very difficult to scroll up and down to
>>>> review it. Uploading it to a Git repository that is browsable in a web
>>>> browser somewhere would be a lot better and we can put any patches on top
>>>> of the branch, so that we only will have smaller changes to review and not
>>>> a whole patchset again and again.
>>>> 
>>> I think the very large patch which gave you problems in May 2020 and linked 
>>> to below and (PATCH v2 0/8 on Patchwork), patched the abandoned version 1 of 
>>> ipblacklist to version 2. The resulting V2 patches:
>>> 
>>> ipblacklist: Build infrastructure
>>> ipblacklist: Modifications to system
>>> ipblacklist: Ancillary files
>>> ipblacklist: WUI menus, language file etc
>>> ipblacklist: WUI Log details page
>>> ipblacklist: WUI Log page
>>> ipblacklist: WUI Settings page
>>> ipblacklist: Main script
>>> 
>>> are on:
>>> https://git.ipfire.org/?p=people/timf/ipfire-2.x.git;a=shortlog;h=refs/heads/ipblacklist
>>> 
>>> I think these were (at May 2020) pretty much ready to go and would only need 
>>> updating if the core files have changed in the meantime. To date the only 
>>> required patch I am aware of would be to ipblacklists.dat to reflect the 
>>> changes in the latest version of IPFire. (There could be others as I only 
>>> patched  my system files rather than replaced them)
>> 
>> What changes are those?
>> 
>>>> Do you have a Git repository somewhere? Would you like me to set up your
>>>> IPFire account so that you can use our servers?
>>>> 
>>> 
>>> I have a Git Development Environment on this computer (Debian Sid 5.16.0-1-
>>> amd64 16GB memory) currently at core163 but is a bit short of room as the 
>>> disk is showing 95% full. Your offer of an account on IPFire might be a 
>>> sensible alternative. 
>> 
>> See above.
>> 
>>>> Do you have experience with Git?
>>>> 
>>>> We would need to rebase the branch onto next (which Adolf has already
>>>> pointed out), but I don’t think this would be a problem because we are
>>>> mainly adding new code and don’t modify too much existing stuff here.
>>>> 
>>> I don't think that would be a problem.
>> 
>> Great!
>> 
>>> 
>>>>> You may be interested in one of the modification I have made to
>>>>> ipblacklist, is to add an additional local blacklist to the sources file
>>>>> to get a
>>>>> blocklist from a web server on my local network.  This is populated by a
>>>>> script which greps the mail server logs for SMTP Auth attacks and has
>>>>> been particularly useful in protecting the mail server from a recent
>>>>> botnet attack where the offending ip addresses have been recycled every
>>>>> one to three weeks. Currently the blocklist contains about 3000 ip
>>>>> addresses and has blocked nearly 2000 smtp auth attempts so far to-day.
>>>>> 
>>>>> I also use fail2ban and Banish to manage iptables blocks on the firewall.
>>>> 
>>>> This is kind of a fail2ban but on the firewall. Since this patchset is
>>>> already so large and there has been a custom blocklist existing before
>>>> which got removed, I would push this project back a little bit until we
>>>> have a base that we can add new features to.
>>>> 
>>> I wasn't suggesting that we add this to the current ipblacklist version. 
>>> However it only requires 1 extra resource added to the sources file so the 
>>> modifications are minimal but would have limited user appeal and it does 
>>> require some local processing of server lo files. However for my botnet 
>>> attack it is being very effective.
>> 
>> Okay. It is great to see that this works wonders already :)
>> 
>>>> I am not entirely convinced that this functionality scales well across all
>>>> users. How would they move the logs to the firewall? We don’t have a
>>>> simple API, but if we did, we would not have a system to authenticated
>>>> other servers. This would be a project that I would find a little bit more
>>>> complicated and we would need a couple more pieces in the puzzle before we
>>>> are ready for it.
>>>> 
>>> I fully agree.
> 
> This sounds as if it does the same sort of thing as something I had in my first patchset.  I added an extra rule to the input policy chain that added the address to an ipset if the number of dropped packets exceeded a threshold.  This runs completely within iptables/ipset.
> 
> iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist src -j SET --add-set $autoblacklist src --exist" );
> iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist dst -j SET --add-set $autoblacklist dst --exist" );
> iptables( "-I POLICYIN 1 -i $red_iface -m hashlimit --hashlimit-mode srcip --hashlimit-above $settings{BLOCK_THRESHOLD}/hour --hashlimit-name $autoblacklist -j SET --add-set $autoblacklist src" );
> 
>>> 
>>>>> The last communication I could find between yourself and Tim was in May
>>>>> 2020. https://lists.ipfire.org/pipermail/development/2020-May/007822.html
>>>> 
>>>> Thanks for finding this. Indeed the conversation just ended in nothingness
>>>> due to lack of time of everybody I would suspect.
>>>> 
>>>> I could not find anything on the list that I would consider a blocker.
>>>> There are however some smaller things like translations and probably there
>>>> will be a couple of minor bugs and some improvements to the look and feel.
>>>> 
>>> There are a couple of points we need to consider:
>>> 
>>> 1) IPBlacklist does not work very well if Tim's  ipfblocklist add-on is also 
>>> installed. My view is that the add-on should be removed before IPBlacklist 
>>> can be applied. Can the add-on be automatically removed on installaion and 
>>> should we transfer the settings info from ipfbocklist to ipblacklist?
>> 
>> Yes, in theory we could remove any old files in the updater and install our own ones.
>> 
>>> 2) I added a init script to my firewall which doesn't seem to be present on 
>>> Tim's patches. I'm not sure if this is needed as it will be started by fcron 
>>> or changes made in the WUI but won't be instantly available on re-boot. Do 
>>> you have any thoughts on this?
>> 
> 
> I don't think this is needed - the change to the firewall init script should call the ipblacklist script at the correct time.
> 
>> What does the script do?
>> 
>>>> So, can we start with rebasing the Git branch, please?
>>>> 
>>> 
>>> No problem.
>>> 
>>> Rob
>>> 
>> 
>> -Michael
> 
> Tim

Re: ipblacklist V2

[Rob Brewer]

[-- Attachment #1: Type: text/plain, Size: 2326 bytes --]

Hi Tim, Good to see your posing here.

On Saturday 12 February 2022 21:29 Tim FitzGeorge wrote:

> This sounds as if it does the same sort of thing as something I had in my first patchset.  I added an extra rule to the input policy chain that added the address to an ipset if the number of dropped packets exceeded a 
threshold.  This runs completely within iptables/ipset.
> 
> iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist src -j SET --add-set $autoblacklist src --exist" );
> iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist dst -j SET --add-set $autoblacklist dst --exist" );
> iptables( "-I POLICYIN 1 -i $red_iface -m hashlimit --hashlimit-mode srcip --hashlimit-above $settings{BLOCK_THRESHOLD}/hour --hashlimit-name $autoblacklist -j SET --add-set $autoblacklist src" );
> 

Aotoblacklist looks like a useful addition. Why did you drop t?


>>>>
>>> There are a couple of points we need to consider:
>>>
>>> 1) IPBlacklist does not work very well if Tim's  ipfblocklist add-on is also 
>>> installed. My view is that the add-on should be removed before IPBlacklist 
>>> can be applied. Can the add-on be automatically removed on installaion and 
>>> should we transfer the settings info from ipfbocklist to ipblacklist?
>> 
>> Yes, in theory we could remove any old files in the updater and install our own ones.
>> 

There are a couple of errors on your uninstall-blocklist.sh script which leaves
some files behind when it is run. I can send you a patch for this if it is of help.

>>> 2) I added a init script to my firewall which doesn't seem to be present on 
>>> Tim's patches. I'm not sure if this is needed as it will be started by fcron 
>>> or changes made in the WUI but won't be instantly available on re-boot. Do 
>>> you have any thoughts on this?
>> 
> 
> I don't think this is needed - the change to the firewall init script should call the ipblacklist script at the correct time.

I hadn't noticed the last few lines in your firewall init script which my init
script duplicates. So I agree my addition isn't needed.

I have started producing the v3 patches requested by the devs, but apart from the couple 
of changes needed to  ipblacklists.dat I think they will be almost identical to your
v2 patches.

Rob 

Re: ipblacklist V2

[Tim FitzGeorge]

[-- Attachment #1: Type: text/plain, Size: 7373 bytes --]

Hello,

On 10/02/2022 16:48, Michael Tremer wrote:
> Hello Rob,
> 
>> On 10 Feb 2022, at 15:12, Rob Brewer <ipfire-devel(a)grantura.co.uk> wrote:
>>
>> Hi Michael
>>
>> On Thursday 10 February 2022 09:41 Michael Tremer wrote:
>>
>>> Hello Rob,
>>>
>>>
>>> Good to hear that you are in touch. I would like to invite Tim to join the
>>> conversation on here. I am sure he has a couple of thoughts to contribute
>>> and I hope he can find the time.
>>>
>>>
>>> I assume you are talking about this here?
>>>
>>>
>> https://git.ipfire.org/?p=people/timf/ipfire-2.x.git;a=shortlog;h=refs/heads/ipblacklist
>>
>> Yes Tim's link to his ipfire git pages which links your URL.
> 
> Okay, I just set up your very own repository over here:
> 
>   https://git.ipfire.org/?p=people/helix/ipfire-2.x.git;a=summary
> 
> I will send you a private email with your account stuff.
> 
>>>
>>> That would have been one of my first questions having looked at my emails
>>> again: Get the code into some Git repository.
>>>
>>> This is a large patchset and it is very difficult to scroll up and down to
>>> review it. Uploading it to a Git repository that is browsable in a web
>>> browser somewhere would be a lot better and we can put any patches on top
>>> of the branch, so that we only will have smaller changes to review and not
>>> a whole patchset again and again.
>>>
>> I think the very large patch which gave you problems in May 2020 and linked 
>> to below and (PATCH v2 0/8 on Patchwork), patched the abandoned version 1 of 
>> ipblacklist to version 2. The resulting V2 patches:
>>
>> ipblacklist: Build infrastructure
>> ipblacklist: Modifications to system
>> ipblacklist: Ancillary files
>> ipblacklist: WUI menus, language file etc
>> ipblacklist: WUI Log details page
>> ipblacklist: WUI Log page
>> ipblacklist: WUI Settings page
>> ipblacklist: Main script
>>
>> are on:
>> https://git.ipfire.org/?p=people/timf/ipfire-2.x.git;a=shortlog;h=refs/heads/ipblacklist
>>
>> I think these were (at May 2020) pretty much ready to go and would only need 
>> updating if the core files have changed in the meantime. To date the only 
>> required patch I am aware of would be to ipblacklists.dat to reflect the 
>> changes in the latest version of IPFire. (There could be others as I only 
>> patched  my system files rather than replaced them)
> 
> What changes are those?
> 
>>> Do you have a Git repository somewhere? Would you like me to set up your
>>> IPFire account so that you can use our servers?
>>>
>>
>> I have a Git Development Environment on this computer (Debian Sid 5.16.0-1-
>> amd64 16GB memory) currently at core163 but is a bit short of room as the 
>> disk is showing 95% full. Your offer of an account on IPFire might be a 
>> sensible alternative. 
> 
> See above.
> 
>>> Do you have experience with Git?
>>>
>>> We would need to rebase the branch onto next (which Adolf has already
>>> pointed out), but I don’t think this would be a problem because we are
>>> mainly adding new code and don’t modify too much existing stuff here.
>>>
>> I don't think that would be a problem.
> 
> Great!
> 
>>
>>>> You may be interested in one of the modification I have made to
>>>> ipblacklist, is to add an additional local blacklist to the sources file
>>>> to get a
>>>> blocklist from a web server on my local network.  This is populated by a
>>>> script which greps the mail server logs for SMTP Auth attacks and has
>>>> been particularly useful in protecting the mail server from a recent
>>>> botnet attack where the offending ip addresses have been recycled every
>>>> one to three weeks. Currently the blocklist contains about 3000 ip
>>>> addresses and has blocked nearly 2000 smtp auth attempts so far to-day.
>>>>
>>>> I also use fail2ban and Banish to manage iptables blocks on the firewall.
>>>
>>> This is kind of a fail2ban but on the firewall. Since this patchset is
>>> already so large and there has been a custom blocklist existing before
>>> which got removed, I would push this project back a little bit until we
>>> have a base that we can add new features to.
>>>
>> I wasn't suggesting that we add this to the current ipblacklist version. 
>> However it only requires 1 extra resource added to the sources file so the 
>> modifications are minimal but would have limited user appeal and it does 
>> require some local processing of server lo files. However for my botnet 
>> attack it is being very effective.
> 
> Okay. It is great to see that this works wonders already :)
> 
>>> I am not entirely convinced that this functionality scales well across all
>>> users. How would they move the logs to the firewall? We don’t have a
>>> simple API, but if we did, we would not have a system to authenticated
>>> other servers. This would be a project that I would find a little bit more
>>> complicated and we would need a couple more pieces in the puzzle before we
>>> are ready for it.
>>>
>> I fully agree.

This sounds as if it does the same sort of thing as something I had in my first patchset.  I added an extra rule to the input policy chain that added the address to an ipset if the number of dropped packets exceeded a threshold.  This runs completely within iptables/ipset.

iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist src -j SET --add-set $autoblacklist src --exist" );
iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist dst -j SET --add-set $autoblacklist dst --exist" );
iptables( "-I POLICYIN 1 -i $red_iface -m hashlimit --hashlimit-mode srcip --hashlimit-above $settings{BLOCK_THRESHOLD}/hour --hashlimit-name $autoblacklist -j SET --add-set $autoblacklist src" );

>>
>>>> The last communication I could find between yourself and Tim was in May
>>>> 2020. https://lists.ipfire.org/pipermail/development/2020-May/007822.html
>>>
>>> Thanks for finding this. Indeed the conversation just ended in nothingness
>>> due to lack of time of everybody I would suspect.
>>>
>>> I could not find anything on the list that I would consider a blocker.
>>> There are however some smaller things like translations and probably there
>>> will be a couple of minor bugs and some improvements to the look and feel.
>>>
>> There are a couple of points we need to consider:
>>
>> 1) IPBlacklist does not work very well if Tim's  ipfblocklist add-on is also 
>> installed. My view is that the add-on should be removed before IPBlacklist 
>> can be applied. Can the add-on be automatically removed on installaion and 
>> should we transfer the settings info from ipfbocklist to ipblacklist?
> 
> Yes, in theory we could remove any old files in the updater and install our own ones.
> 
>> 2) I added a init script to my firewall which doesn't seem to be present on 
>> Tim's patches. I'm not sure if this is needed as it will be started by fcron 
>> or changes made in the WUI but won't be instantly available on re-boot. Do 
>> you have any thoughts on this?
> 

I don't think this is needed - the change to the firewall init script should call the ipblacklist script at the correct time.

> What does the script do?
> 
>>> So, can we start with rebasing the Git branch, please?
>>>
>>
>> No problem.
>>
>> Rob
>>
> 
> -Michael

Tim

Re: ipblacklist V2

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 6305 bytes --]

Hello Rob,

> On 10 Feb 2022, at 15:12, Rob Brewer <ipfire-devel(a)grantura.co.uk> wrote:
> 
> Hi Michael
> 
> On Thursday 10 February 2022 09:41 Michael Tremer wrote:
> 
>> Hello Rob,
>> 
>> 
>> Good to hear that you are in touch. I would like to invite Tim to join the
>> conversation on here. I am sure he has a couple of thoughts to contribute
>> and I hope he can find the time.
>> 
>> 
>> I assume you are talking about this here?
>> 
>> 
> https://git.ipfire.org/?p=people/timf/ipfire-2.x.git;a=shortlog;h=refs/heads/ipblacklist
> 
> Yes Tim's link to his ipfire git pages which links your URL.

Okay, I just set up your very own repository over here:

  https://git.ipfire.org/?p=people/helix/ipfire-2.x.git;a=summary

I will send you a private email with your account stuff.

>> 
>> That would have been one of my first questions having looked at my emails
>> again: Get the code into some Git repository.
>> 
>> This is a large patchset and it is very difficult to scroll up and down to
>> review it. Uploading it to a Git repository that is browsable in a web
>> browser somewhere would be a lot better and we can put any patches on top
>> of the branch, so that we only will have smaller changes to review and not
>> a whole patchset again and again.
>> 
> I think the very large patch which gave you problems in May 2020 and linked 
> to below and (PATCH v2 0/8 on Patchwork), patched the abandoned version 1 of 
> ipblacklist to version 2. The resulting V2 patches:
> 
> ipblacklist: Build infrastructure
> ipblacklist: Modifications to system
> ipblacklist: Ancillary files
> ipblacklist: WUI menus, language file etc
> ipblacklist: WUI Log details page
> ipblacklist: WUI Log page
> ipblacklist: WUI Settings page
> ipblacklist: Main script
> 
> are on:
> https://git.ipfire.org/?p=people/timf/ipfire-2.x.git;a=shortlog;h=refs/heads/ipblacklist
> 
> I think these were (at May 2020) pretty much ready to go and would only need 
> updating if the core files have changed in the meantime. To date the only 
> required patch I am aware of would be to ipblacklists.dat to reflect the 
> changes in the latest version of IPFire. (There could be others as I only 
> patched  my system files rather than replaced them)

What changes are those?

>> Do you have a Git repository somewhere? Would you like me to set up your
>> IPFire account so that you can use our servers?
>> 
> 
> I have a Git Development Environment on this computer (Debian Sid 5.16.0-1-
> amd64 16GB memory) currently at core163 but is a bit short of room as the 
> disk is showing 95% full. Your offer of an account on IPFire might be a 
> sensible alternative. 

See above.

>> Do you have experience with Git?
>> 
>> We would need to rebase the branch onto next (which Adolf has already
>> pointed out), but I don’t think this would be a problem because we are
>> mainly adding new code and don’t modify too much existing stuff here.
>> 
> I don't think that would be a problem.

Great!

> 
>>> You may be interested in one of the modification I have made to
>>> ipblacklist, is to add an additional local blacklist to the sources file
>>> to get a
>>> blocklist from a web server on my local network.  This is populated by a
>>> script which greps the mail server logs for SMTP Auth attacks and has
>>> been particularly useful in protecting the mail server from a recent
>>> botnet attack where the offending ip addresses have been recycled every
>>> one to three weeks. Currently the blocklist contains about 3000 ip
>>> addresses and has blocked nearly 2000 smtp auth attempts so far to-day.
>>> 
>>> I also use fail2ban and Banish to manage iptables blocks on the firewall.
>> 
>> This is kind of a fail2ban but on the firewall. Since this patchset is
>> already so large and there has been a custom blocklist existing before
>> which got removed, I would push this project back a little bit until we
>> have a base that we can add new features to.
>> 
> I wasn't suggesting that we add this to the current ipblacklist version. 
> However it only requires 1 extra resource added to the sources file so the 
> modifications are minimal but would have limited user appeal and it does 
> require some local processing of server lo files. However for my botnet 
> attack it is being very effective.

Okay. It is great to see that this works wonders already :)

>> I am not entirely convinced that this functionality scales well across all
>> users. How would they move the logs to the firewall? We don’t have a
>> simple API, but if we did, we would not have a system to authenticated
>> other servers. This would be a project that I would find a little bit more
>> complicated and we would need a couple more pieces in the puzzle before we
>> are ready for it.
>> 
> I fully agree.
> 
>>> The last communication I could find between yourself and Tim was in May
>>> 2020. https://lists.ipfire.org/pipermail/development/2020-May/007822.html
>> 
>> Thanks for finding this. Indeed the conversation just ended in nothingness
>> due to lack of time of everybody I would suspect.
>> 
>> I could not find anything on the list that I would consider a blocker.
>> There are however some smaller things like translations and probably there
>> will be a couple of minor bugs and some improvements to the look and feel.
>> 
> There are a couple of points we need to consider:
> 
> 1) IPBlacklist does not work very well if Tim's  ipfblocklist add-on is also 
> installed. My view is that the add-on should be removed before IPBlacklist 
> can be applied. Can the add-on be automatically removed on installaion and 
> should we transfer the settings info from ipfbocklist to ipblacklist?

Yes, in theory we could remove any old files in the updater and install our own ones.

> 2) I added a init script to my firewall which doesn't seem to be present on 
> Tim's patches. I'm not sure if this is needed as it will be started by fcron 
> or changes made in the WUI but won't be instantly available on re-boot. Do 
> you have any thoughts on this?

What does the script do?

>> So, can we start with rebasing the Git branch, please?
>> 
> 
> No problem.
> 
> Rob
> 

-Michael

Re: ipblacklist V2

[Rob Brewer]

[-- Attachment #1: Type: text/plain, Size: 5603 bytes --]

Hi Michael

On Thursday 10 February 2022 09:41 Michael Tremer wrote:

> Hello Rob,
> 
> 
> Good to hear that you are in touch. I would like to invite Tim to join the
> conversation on here. I am sure he has a couple of thoughts to contribute
> and I hope he can find the time.
> 
> 
> I assume you are talking about this here?
> 
>   
https://git.ipfire.org/?p=people/timf/ipfire-2.x.git;a=shortlog;h=refs/heads/ipblacklist

Yes Tim's link to his ipfire git pages which links your URL.

> 
> That would have been one of my first questions having looked at my emails
> again: Get the code into some Git repository.
> 
> This is a large patchset and it is very difficult to scroll up and down to
> review it. Uploading it to a Git repository that is browsable in a web
> browser somewhere would be a lot better and we can put any patches on top
> of the branch, so that we only will have smaller changes to review and not
> a whole patchset again and again.
> 
I think the very large patch which gave you problems in May 2020 and linked 
to below and (PATCH v2 0/8 on Patchwork), patched the abandoned version 1 of 
ipblacklist to version 2. The resulting V2 patches:

ipblacklist: Build infrastructure
ipblacklist: Modifications to system
ipblacklist: Ancillary files
ipblacklist: WUI menus, language file etc
ipblacklist: WUI Log details page
ipblacklist: WUI Log page
ipblacklist: WUI Settings page
ipblacklist: Main script

are on:
https://git.ipfire.org/?p=people/timf/ipfire-2.x.git;a=shortlog;h=refs/heads/ipblacklist

I think these were (at May 2020) pretty much ready to go and would only need 
updating if the core files have changed in the meantime. To date the only 
required patch I am aware of would be to ipblacklists.dat to reflect the 
changes in the latest version of IPFire. (There could be others as I only 
patched  my system files rather than replaced them)


> Do you have a Git repository somewhere? Would you like me to set up your
> IPFire account so that you can use our servers?
> 

I have a Git Development Environment on this computer (Debian Sid 5.16.0-1-
amd64 16GB memory) currently at core163 but is a bit short of room as the 
disk is showing 95% full. Your offer of an account on IPFire might be a 
sensible alternative. 


> Do you have experience with Git?
> 
> We would need to rebase the branch onto next (which Adolf has already
> pointed out), but I don’t think this would be a problem because we are
> mainly adding new code and don’t modify too much existing stuff here.
> 
I don't think that would be a problem.

>> You may be interested in one of the modification I have made to
>> ipblacklist, is to add an additional local blacklist to the sources file
>> to get a
>> blocklist from a web server on my local network.  This is populated by a
>> script which greps the mail server logs for SMTP Auth attacks and has
>> been particularly useful in protecting the mail server from a recent
>> botnet attack where the offending ip addresses have been recycled every
>> one to three weeks. Currently the blocklist contains about 3000 ip
>> addresses and has blocked nearly 2000 smtp auth attempts so far to-day.
>> 
>> I also use fail2ban and Banish to manage iptables blocks on the firewall.
> 
> This is kind of a fail2ban but on the firewall. Since this patchset is
> already so large and there has been a custom blocklist existing before
> which got removed, I would push this project back a little bit until we
> have a base that we can add new features to.
> 
I wasn't suggesting that we add this to the current ipblacklist version. 
However it only requires 1 extra resource added to the sources file so the 
modifications are minimal but would have limited user appeal and it does 
require some local processing of server lo files. However for my botnet 
attack it is being very effective.

> I am not entirely convinced that this functionality scales well across all
> users. How would they move the logs to the firewall? We don’t have a
> simple API, but if we did, we would not have a system to authenticated
> other servers. This would be a project that I would find a little bit more
> complicated and we would need a couple more pieces in the puzzle before we
> are ready for it.
> 
I fully agree.

>> The last communication I could find between yourself and Tim was in May
>> 2020. https://lists.ipfire.org/pipermail/development/2020-May/007822.html
> 
> Thanks for finding this. Indeed the conversation just ended in nothingness
> due to lack of time of everybody I would suspect.
> 
> I could not find anything on the list that I would consider a blocker.
> There are however some smaller things like translations and probably there
> will be a couple of minor bugs and some improvements to the look and feel.
> 
There are a couple of points we need to consider:

1) IPBlacklist does not work very well if Tim's  ipfblocklist add-on is also 
installed. My view is that the add-on should be removed before IPBlacklist 
can be applied. Can the add-on be automatically removed on installaion and 
should we transfer the settings info from ipfbocklist to ipblacklist?

2) I added a init script to my firewall which doesn't seem to be present on 
Tim's patches. I'm not sure if this is needed as it will be started by fcron 
or changes made in the WUI but won't be instantly available on re-boot. Do 
you have any thoughts on this?


> So, can we start with rebasing the Git branch, please?
> 

No problem.

Rob

Re: ipblacklist V2

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 3600 bytes --]

Hello,

> On 9 Feb 2022, at 22:14, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Rob,
> 
> On 09/02/2022 22:43, Rob Brewer wrote:
>> Hi Adolf,
>> 
>> Thank you for the kind comments in your earlier post and I'm pleased that
>> ipblacklist will be incorporated into IPFire. From what I'm seeing here it
>> will be a very worthwhile addition.  I have been looking through the Wiki
>> documentation for submitting patches and it looks fairly explanatory but
>> will probably need your help  to keep me in the right place.
> Just ask when you have any questions.
>> Adolf Belka wrote:
>> 
>>> Hi Rob,
>>> 
>>> In your communication below about the various patches, it sounded to me
>>> like you are suggesting that some of the patches could be taken from the
>>> earlier patchset because they have not changed. My apologies if I have
>>> misread it.
>>> 
>> I think that is correct. The majority of Tim's V2 patches can be
>> incorporated without changes.
>> 
>>> I believe that it would be better to submit a complete patchset now, where
>>> you include all patches, even if they haven't changed from earlier. This
>>> would make it much easier for tracking the status of the patches within
>>> patchwork and knowing what was included at a later date if looking back.
>>> So Tim's old patchset would be marked as superseded in patchwork and yours
>>> would take over and would be named as a v3 version.
>>> 
>> That's OK but it'll get me a couple of days to get my head around submitting
>> patches and getting the correct build environment set up. I presume you
>> would need patches against the next development load
>> (git checkout -b next -t origin/next) which I will have to set up.
> That is correct.
>>> Information on submitting the patches is in the wiki and if needed I can
>>> provide the appropriate links. If you have any questions on or problems
>>> with the patch submission process then just let me know.
>>> 
>> Thanks Adolf, I've read some of the docs. It mostly looks OK but when I get
>> around to submitting patches I think I'll need some help on submitting the
>> Build infrastructure patches and I don't have a clue how to generate the
>> lang files.
> 
> For the languages that are not provided then the default in IPFire is that the English version will be used. This is also the case if some but not all entries are translated. The untranslated entries stay as English.If there are languages where there are no files then you don't have to worry about them.  IPFire users who are native speakers usually over time provide the translations, either as patches or as files that one of us can create patches from.

Let’s not worry about any other language than English for now. Everyone on the team can handle this during the development process and we should translate everything in the end. Otherwise we might change a string or add another checkbox somewhere and we need to reach out to the native speakers of the other languages for just one sentence.

We have lots of native German speakers on here, so that won’t be a problem. Stéphane is a native French speaker who regularly updates the French translation and to whom we should reach out to when the time is right :) Otherwise the language support in IPFire is pretty much best-effort only.

-Michael

> 
> I will send you tomorrow the commands you need to run for the language files after the build command has been completed.
> 
>> It looks like the first step will be difficult but then hopefully gets
>> easier.
>> 
>> Regards
>> 
>> Rob

Re: ipblacklist V2

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 4125 bytes --]

Hello Rob,

> On 9 Feb 2022, at 13:23, Rob Brewer <ipfire-devel(a)grantura.co.uk> wrote:
> 
> Hi Michael
> 
> 
> Michael Tremer wrote:
> 
>> Hello Rob,
>> 
>> Thank you for your interest in working on this.
>> 
>> Yes, I always thought that there was great interest in moving this over
>> the line. However, I could not find where we left off here.
>> 
>> There were a couple of outstanding issues that had to be resolved. I just
>> couldn’t find my last emails. Are you aware of these?
>> 
> 
> I'm pleased to say I have had an email from Tim and is supportive of my 
> attempts to progress ipblacklist into IPFire. Tim however says "Between 
> COVID, my taking on additional responsibilities and the code not being part 
> of ipfire, it's currently got a very low priority for me."

Good to hear that you are in touch. I would like to invite Tim to join the conversation on here. I am sure he has a couple of thoughts to contribute and I hope he can find the time.

> Tim pointed me to his git pages where I was able to find most of the code 
> that I thought was missing from patchwork and is all now installed on my 
> firewall and is working extremely well.

I assume you are talking about this here?

  https://git.ipfire.org/?p=people/timf/ipfire-2.x.git;a=shortlog;h=refs/heads/ipblacklist

That would have been one of my first questions having looked at my emails again: Get the code into some Git repository.

This is a large patchset and it is very difficult to scroll up and down to review it. Uploading it to a Git repository that is browsable in a web browser somewhere would be a lot better and we can put any patches on top of the branch, so that we only will have smaller changes to review and not a whole patchset again and again.

Do you have a Git repository somewhere? Would you like me to set up your IPFire account so that you can use our servers?

Do you have experience with Git?

We would need to rebase the branch onto next (which Adolf has already pointed out), but I don’t think this would be a problem because we are mainly adding new code and don’t modify too much existing stuff here.

> You may be interested in one of the modification I have made to ipblacklist, 
> is to add an additional local blacklist to the sources file to get a 
> blocklist from a web server on my local network.  This is populated by a 
> script which greps the mail server logs for SMTP Auth attacks and has been  
> particularly useful in protecting the mail server from a recent botnet 
> attack where the offending ip addresses have been recycled every one to 
> three weeks. Currently the blocklist contains about 3000 ip addresses and 
> has blocked nearly 2000 smtp auth attempts so far to-day.
> 
> I also use fail2ban and Banish to manage iptables blocks on the firewall.

This is kind of a fail2ban but on the firewall. Since this patchset is already so large and there has been a custom blocklist existing before which got removed, I would push this project back a little bit until we have a base that we can add new features to.

I am not entirely convinced that this functionality scales well across all users. How would they move the logs to the firewall? We don’t have a simple API, but if we did, we would not have a system to authenticated other servers. This would be a project that I would find a little bit more complicated and we would need a couple more pieces in the puzzle before we are ready for it.

> The last communication I could find between yourself and Tim was in May 
> 2020. https://lists.ipfire.org/pipermail/development/2020-May/007822.html

Thanks for finding this. Indeed the conversation just ended in nothingness due to lack of time of everybody I would suspect.

I could not find anything on the list that I would consider a blocker. There are however some smaller things like translations and probably there will be a couple of minor bugs and some improvements to the look and feel.

So, can we start with rebasing the Git branch, please?

-Michael

> Hope this is useful.
> 
> Rob
> 
> 

Re: ipblacklist V2

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 2780 bytes --]

Hi Rob,

On 09/02/2022 22:43, Rob Brewer wrote:
> Hi Adolf,
>
> Thank you for the kind comments in your earlier post and I'm pleased that
> ipblacklist will be incorporated into IPFire. From what I'm seeing here it
> will be a very worthwhile addition.  I have been looking through the Wiki
> documentation for submitting patches and it looks fairly explanatory but
> will probably need your help  to keep me in the right place.
Just ask when you have any questions.
> Adolf Belka wrote:
>
>> Hi Rob,
>>
>> In your communication below about the various patches, it sounded to me
>> like you are suggesting that some of the patches could be taken from the
>> earlier patchset because they have not changed. My apologies if I have
>> misread it.
>>
> I think that is correct. The majority of Tim's V2 patches can be
> incorporated without changes.
>
>> I believe that it would be better to submit a complete patchset now, where
>> you include all patches, even if they haven't changed from earlier. This
>> would make it much easier for tracking the status of the patches within
>> patchwork and knowing what was included at a later date if looking back.
>> So Tim's old patchset would be marked as superseded in patchwork and yours
>> would take over and would be named as a v3 version.
>>
> That's OK but it'll get me a couple of days to get my head around submitting
> patches and getting the correct build environment set up. I presume you
> would need patches against the next development load
> (git checkout -b next -t origin/next) which I will have to set up.
That is correct.
>> Information on submitting the patches is in the wiki and if needed I can
>> provide the appropriate links. If you have any questions on or problems
>> with the patch submission process then just let me know.
>>
> Thanks Adolf, I've read some of the docs. It mostly looks OK but when I get
> around to submitting patches I think I'll need some help on submitting the
> Build infrastructure patches and I don't have a clue how to generate the
> lang files.

For the languages that are not provided then the default in IPFire is that the English version will be used. This is also the case if some but not all entries are translated. The untranslated entries stay as English.If there are languages where there are no files then you don't have to worry about them.  IPFire users who are native speakers usually over time provide the translations, either as patches or as files that one of us can create patches from.

I will send you tomorrow the commands you need to run for the language files after the build command has been completed.

> It looks like the first step will be difficult but then hopefully gets
> easier.
>
> Regards
>
> Rob

Re: ipblacklist V2

[Rob Brewer]

[-- Attachment #1: Type: text/plain, Size: 1995 bytes --]

Hi Adolf,

Thank you for the kind comments in your earlier post and I'm pleased that 
ipblacklist will be incorporated into IPFire. From what I'm seeing here it 
will be a very worthwhile addition.  I have been looking through the Wiki 
documentation for submitting patches and it looks fairly explanatory but 
will probably need your help  to keep me in the right place.

Adolf Belka wrote:

> Hi Rob,
> 
> In your communication below about the various patches, it sounded to me
> like you are suggesting that some of the patches could be taken from the
> earlier patchset because they have not changed. My apologies if I have
> misread it.
> 
I think that is correct. The majority of Tim's V2 patches can be 
incorporated without changes.

> I believe that it would be better to submit a complete patchset now, where
> you include all patches, even if they haven't changed from earlier. This
> would make it much easier for tracking the status of the patches within
> patchwork and knowing what was included at a later date if looking back.
> So Tim's old patchset would be marked as superseded in patchwork and yours
> would take over and would be named as a v3 version.
> 
That's OK but it'll get me a couple of days to get my head around submitting 
patches and getting the correct build environment set up. I presume you 
would need patches against the next development load 
(git checkout -b next -t origin/next) which I will have to set up.

> Information on submitting the patches is in the wiki and if needed I can
> provide the appropriate links. If you have any questions on or problems
> with the patch submission process then just let me know.
> 
Thanks Adolf, I've read some of the docs. It mostly looks OK but when I get 
around to submitting patches I think I'll need some help on submitting the 
Build infrastructure patches and I don't have a clue how to generate the 
lang files.

It looks like the first step will be difficult but then hopefully gets 
easier.

Regards

Rob

Re: ipblacklist V2

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 4411 bytes --]

Hi Rob,

In your communication below about the various patches, it sounded to me like you are suggesting that some of the patches could be taken from the earlier patchset because they have not changed. My apologies if I have misread it.

I believe that it would be better to submit a complete patchset now, where you include all patches, even if they haven't changed from earlier.
This would make it much easier for tracking the status of the patches within patchwork and knowing what was included at a later date if looking back. So Tim's old patchset would be marked as superseded in patchwork and yours would take over and would be named as a v3 version.

Information on submitting the patches is in the wiki and if needed I can provide the appropriate links. If you have any questions on or problems with the patch submission process then just let me know.

Regards,

Adolf.

On 07/02/2022 13:58, Rob Brewer wrote:
> Hi Michael,
>
>
> Michael Tremer wrote:
>
>> Hello Rob,
>>
>> Thank you for your interest in working on this.
>>
>> Yes, I always thought that there was great interest in moving this over
>> the line. However, I could not find where we left off here.
>>
>> There were a couple of outstanding issues that had to be resolved. I just
>> couldn’t find my last emails. Are you aware of these?
>>
> You expect have seen the topic :
>
> https://community.ipfire.org/t/can-i-help-progress-version2-of-ipblacklist-addon/6998/18
>
>   where Adolf has been encouraging me to look at this feature and I now have
> a working copy on my firewall.
>
> I believe the patches on patchwork are mostly complete and with a few
> additions/corrections can be incorporated into the latest build.
>
> Here is my understanding of where we are:
>
> https://patchwork.ipfire.org/project/ipfire/patch/20200427143123.6378-9-ipfr(a)tfitzgeorge.me.uk/
>
> [v2,0/8] ipblacklist: IP Address Blacklists
>
> This patch was applied to convert V1 into V2 and used to make the the
> remaining patches
> It can safely be ignored.
>
> [v2,1/8] ipblacklist: Main script
> This is the ipblacklist main perl program. It can be incorporated without
> problems.
>
>
> [v2,2/8] ipblacklist: WUI Settings page
> ipblacklist.cgi Patch can be applied without any changes.
>
>
> [v2,3/8] ipblacklist: WUI Log page
> ipblacklists.dat  - Needs to be modified with an additional patch to:
>
> 1)    remove - require “${General::swroot}/geoip-functions.pl”; - This isn’t
> included in core 161 and not used in this program.
>
> 2)  change
>          &General::readhash( “/srv/web/ipfire/html/themes/”.
> $mainsettings{‘THEME’}."/include/colors.txt", %color );
>      to
>          &General::readhash(
> “/srv/web/ipfire/html/themes/ipfire/include/colors.txt”, %color );
>
>
> [v2,4/8] ipblacklist: WUI Log details page
>
> showrequestfromblacklist.dat - Patch can be applied without any changes
>
>
> [v2,5/8] ipblacklist: WUI menus, language file etc
>
> 50-firewall.menu
> 70-log.menu
> log.dat
> en.pl
>
> Requires additional lang files for de, es, fr, it, nl, pl, ru, tr.
> All others are OK
>
> [v2,6/8] ipblacklist: Ancillary files
>
> sources
> getipsetstat.c
> ipblacklistctrl.c
>
>   Can be applied without any changes
>   
>
> [v2,8/8] ipblacklist: Build infrastructure
>
> aarch64/stage2
> configroot
> ipblacklist-sources
> logwatch
> misc-progs
> common/stage2
> web-user-interface
> x86_64/stage2
> lfs/configroot
> lfs/ipblacklist-sources
> lfs/logwatch
> make.sh
> misc-progs/Makefile
>
> Missing Files and links:
>
> ipblacklist.conf (for logwatch)
> ipblacklist (for logwatch)
> /etc/rc.d/init.d/ipblacklist (Start -up)
> /etc/rc.d/rcsysinit.d/S87ipblacklist -> ../init.d/ipblacklist
>
> I have written code for the missing files which are working on my firewall
> and I could submit as patches if required.
>
>   /usr/share/logwatch/default.conf/services/ipblacklist.conf
>   /usr/share/logwatch/scripts/services/ipblacklist
>   /etc/rc.d/init.d/ipblacklist
>
> I haven't tried incorporating the patches as I extracted the files from
> Patchwork and manually inserted them into my firewall for testing. I would
> be very surprised if there were any problems  with Tim's code however.
>
> Hope this will help to progress this very useful addition to IPFire.
>
>
> Rob
>

Re: ipblacklist V2

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 2722 bytes --]

Hi Rob,
Good to hear that you had some contact with Tim and he is supportive of you picking this up.

Just to let you know that this ipblacklist was discussed at the monthly IPFire Development Conf call last Monday and your work on this was appreciated. The decision was that it should be a core part of IPFire and not an addon. The view is that IPFire as a firewall should have everything required for a firewall available as default. The only things that should be addons are those that are not a core part of a firewall. An ipblacklist is something that should be available to the firewall as standard.
It was also agreed that I should help support you in whatever way I can for the patch submission etc, if required.
I will separately reply to your earlier e-mail about the various patches.
Michael can give any comment he wants with regard to the earlier communications.

Regards,

Adolf.


On 09/02/2022 14:23, Rob Brewer wrote:
> Hi Michael
>
>
> Michael Tremer wrote:
>
>> Hello Rob,
>>
>> Thank you for your interest in working on this.
>>
>> Yes, I always thought that there was great interest in moving this over
>> the line. However, I could not find where we left off here.
>>
>> There were a couple of outstanding issues that had to be resolved. I just
>> couldn’t find my last emails. Are you aware of these?
>>
> I'm pleased to say I have had an email from Tim and is supportive of my
> attempts to progress ipblacklist into IPFire. Tim however says "Between
> COVID, my taking on additional responsibilities and the code not being part
> of ipfire, it's currently got a very low priority for me."
>
> Tim pointed me to his git pages where I was able to find most of the code
> that I thought was missing from patchwork and is all now installed on my
> firewall and is working extremely well.
>
> You may be interested in one of the modification I have made to ipblacklist,
> is to add an additional local blacklist to the sources file to get a
> blocklist from a web server on my local network.  This is populated by a
> script which greps the mail server logs for SMTP Auth attacks and has been
> particularly useful in protecting the mail server from a recent botnet
> attack where the offending ip addresses have been recycled every one to
> three weeks. Currently the blocklist contains about 3000 ip addresses and
> has blocked nearly 2000 smtp auth attempts so far to-day.
>
> I also use fail2ban and Banish to manage iptables blocks on the firewall.
>
> The last communication I could find between yourself and Tim was in May
> 2020. https://lists.ipfire.org/pipermail/development/2020-May/007822.html
>
> Hope this is useful.
>
> Rob
>
>

Re: ipblacklist V2

[Rob Brewer]

[-- Attachment #1: Type: text/plain, Size: 1684 bytes --]

Hi Michael


Michael Tremer wrote:

> Hello Rob,
> 
> Thank you for your interest in working on this.
> 
> Yes, I always thought that there was great interest in moving this over
> the line. However, I could not find where we left off here.
> 
> There were a couple of outstanding issues that had to be resolved. I just
> couldn’t find my last emails. Are you aware of these?
> 

I'm pleased to say I have had an email from Tim and is supportive of my 
attempts to progress ipblacklist into IPFire. Tim however says "Between 
COVID, my taking on additional responsibilities and the code not being part 
of ipfire, it's currently got a very low priority for me."

Tim pointed me to his git pages where I was able to find most of the code 
that I thought was missing from patchwork and is all now installed on my 
firewall and is working extremely well.

You may be interested in one of the modification I have made to ipblacklist, 
is to add an additional local blacklist to the sources file to get a 
blocklist from a web server on my local network.  This is populated by a 
script which greps the mail server logs for SMTP Auth attacks and has been  
particularly useful in protecting the mail server from a recent botnet 
attack where the offending ip addresses have been recycled every one to 
three weeks. Currently the blocklist contains about 3000 ip addresses and 
has blocked nearly 2000 smtp auth attempts so far to-day.

I also use fail2ban and Banish to manage iptables blocks on the firewall.

The last communication I could find between yourself and Tim was in May 
2020. https://lists.ipfire.org/pipermail/development/2020-May/007822.html

Hope this is useful.

Rob

Re: ipblacklist V2

[Rob Brewer]

[-- Attachment #1: Type: text/plain, Size: 3293 bytes --]

Hi Michael,


Michael Tremer wrote:

> Hello Rob,
> 
> Thank you for your interest in working on this.
> 
> Yes, I always thought that there was great interest in moving this over
> the line. However, I could not find where we left off here.
> 
> There were a couple of outstanding issues that had to be resolved. I just
> couldn’t find my last emails. Are you aware of these?
> 

You expect have seen the topic :

https://community.ipfire.org/t/can-i-help-progress-version2-of-ipblacklist-addon/6998/18

 where Adolf has been encouraging me to look at this feature and I now have 
a working copy on my firewall.

I believe the patches on patchwork are mostly complete and with a few 
additions/corrections can be incorporated into the latest build.

Here is my understanding of where we are:

https://patchwork.ipfire.org/project/ipfire/patch/20200427143123.6378-9-ipfr(a)tfitzgeorge.me.uk/

[v2,0/8] ipblacklist: IP Address Blacklists

This patch was applied to convert V1 into V2 and used to make the the 
remaining patches
It can safely be ignored.

[v2,1/8] ipblacklist: Main script
This is the ipblacklist main perl program. It can be incorporated without 
problems.


[v2,2/8] ipblacklist: WUI Settings page
ipblacklist.cgi Patch can be applied without any changes.


[v2,3/8] ipblacklist: WUI Log page
ipblacklists.dat  - Needs to be modified with an additional patch to:

1)    remove - require “${General::swroot}/geoip-functions.pl”; - This isn’t 
included in core 161 and not used in this program.

2)  change
        &General::readhash( “/srv/web/ipfire/html/themes/”.
$mainsettings{‘THEME’}."/include/colors.txt", %color );
    to
        &General::readhash( 
“/srv/web/ipfire/html/themes/ipfire/include/colors.txt”, %color );


[v2,4/8] ipblacklist: WUI Log details page

showrequestfromblacklist.dat - Patch can be applied without any changes


[v2,5/8] ipblacklist: WUI menus, language file etc

50-firewall.menu
70-log.menu
log.dat
en.pl

Requires additional lang files for de, es, fr, it, nl, pl, ru, tr.
All others are OK

[v2,6/8] ipblacklist: Ancillary files

sources
getipsetstat.c 
ipblacklistctrl.c

 Can be applied without any changes
 

[v2,8/8] ipblacklist: Build infrastructure

aarch64/stage2
configroot
ipblacklist-sources
logwatch
misc-progs
common/stage2
web-user-interface
x86_64/stage2
lfs/configroot
lfs/ipblacklist-sources
lfs/logwatch
make.sh
misc-progs/Makefile

Missing Files and links:

ipblacklist.conf (for logwatch)
ipblacklist (for logwatch)
/etc/rc.d/init.d/ipblacklist (Start -up)
/etc/rc.d/rcsysinit.d/S87ipblacklist -> ../init.d/ipblacklist

I have written code for the missing files which are working on my firewall 
and I could submit as patches if required. 

 /usr/share/logwatch/default.conf/services/ipblacklist.conf
 /usr/share/logwatch/scripts/services/ipblacklist
 /etc/rc.d/init.d/ipblacklist 

I haven't tried incorporating the patches as I extracted the files from 
Patchwork and manually inserted them into my firewall for testing. I would 
be very surprised if there were any problems  with Tim's code however.

Hope this will help to progress this very useful addition to IPFire.


Rob

Re: ipblacklist V2

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1143 bytes --]

Hello Rob,

Thank you for your interest in working on this.

Yes, I always thought that there was great interest in moving this over the line. However, I could not find where we left off here.

There were a couple of outstanding issues that had to be resolved. I just couldn’t find my last emails. Are you aware of these?

-Michael

> On 4 Feb 2022, at 18:17, Rob Brewer <ipfire-devel(a)grantura.co.uk> wrote:
> 
> Hi.
> 
> I have been looking at Tim FitzGeorge's code for ipblacklist v2 on 
> https://patchwork.ipfire.org/project/ipfire/list/?series=1215 to see if I 
> can help progress its incorporation into IPFire.
> After I extracted the programs from Patchwork I have been able to build them 
> into my firewall where they are running very successfully.
> The code on the server seems to be in good shape and apart from a few small 
> patches and additions of a few missing scripts I think it could be 
> successfully introduced into the IPFire code base.
> I am more than happy to help in seeing this process carried out but need to 
> know if this is acceptable to yourselves.
> 
> Regards
> Rob Brewer

ipblacklist V2

[Rob Brewer]

[-- Attachment #1: Type: text/plain, Size: 669 bytes --]

Hi.

I have been looking at Tim FitzGeorge's code for ipblacklist v2 on 
https://patchwork.ipfire.org/project/ipfire/list/?series=1215 to see if I 
can help progress its incorporation into IPFire.
After I extracted the programs from Patchwork I have been able to build them 
into my firewall where they are running very successfully.
The code on the server seems to be in good shape and apart from a few small 
patches and additions of a few missing scripts I think it could be 
successfully introduced into the IPFire code base.
I am more than happy to help in seeing this process carried out but need to 
know if this is acceptable to yourselves.

Regards
Rob Brewer