Hi.
I have been looking at Tim FitzGeorge's code for ipblacklist v2 on https://patchwork.ipfire.org/project/ipfire/list/?series=1215 to see if I can help progress its incorporation into IPFire. After I extracted the programs from Patchwork I have been able to build them into my firewall where they are running very successfully. The code on the server seems to be in good shape and apart from a few small patches and additions of a few missing scripts I think it could be successfully introduced into the IPFire code base. I am more than happy to help in seeing this process carried out but need to know if this is acceptable to yourselves.
Regards Rob Brewer
Hello Rob,
Thank you for your interest in working on this.
Yes, I always thought that there was great interest in moving this over the line. However, I could not find where we left off here.
There were a couple of outstanding issues that had to be resolved. I just couldn’t find my last emails. Are you aware of these?
-Michael
On 4 Feb 2022, at 18:17, Rob Brewer ipfire-devel@grantura.co.uk wrote:
Hi.
I have been looking at Tim FitzGeorge's code for ipblacklist v2 on https://patchwork.ipfire.org/project/ipfire/list/?series=1215 to see if I can help progress its incorporation into IPFire. After I extracted the programs from Patchwork I have been able to build them into my firewall where they are running very successfully. The code on the server seems to be in good shape and apart from a few small patches and additions of a few missing scripts I think it could be successfully introduced into the IPFire code base. I am more than happy to help in seeing this process carried out but need to know if this is acceptable to yourselves.
Regards Rob Brewer
Hi Michael,
Michael Tremer wrote:
Hello Rob,
Thank you for your interest in working on this.
Yes, I always thought that there was great interest in moving this over the line. However, I could not find where we left off here.
There were a couple of outstanding issues that had to be resolved. I just couldn’t find my last emails. Are you aware of these?
You expect have seen the topic :
https://community.ipfire.org/t/can-i-help-progress-version2-of-ipblacklist-a...
where Adolf has been encouraging me to look at this feature and I now have a working copy on my firewall.
I believe the patches on patchwork are mostly complete and with a few additions/corrections can be incorporated into the latest build.
Here is my understanding of where we are:
https://patchwork.ipfire.org/project/ipfire/patch/20200427143123.6378-9-ipfr...
[v2,0/8] ipblacklist: IP Address Blacklists
This patch was applied to convert V1 into V2 and used to make the the remaining patches It can safely be ignored.
[v2,1/8] ipblacklist: Main script This is the ipblacklist main perl program. It can be incorporated without problems.
[v2,2/8] ipblacklist: WUI Settings page ipblacklist.cgi Patch can be applied without any changes.
[v2,3/8] ipblacklist: WUI Log page ipblacklists.dat - Needs to be modified with an additional patch to:
1) remove - require “${General::swroot}/geoip-functions.pl”; - This isn’t included in core 161 and not used in this program.
2) change &General::readhash( “/srv/web/ipfire/html/themes/”. $mainsettings{‘THEME’}."/include/colors.txt", %color ); to &General::readhash( “/srv/web/ipfire/html/themes/ipfire/include/colors.txt”, %color );
[v2,4/8] ipblacklist: WUI Log details page
showrequestfromblacklist.dat - Patch can be applied without any changes
[v2,5/8] ipblacklist: WUI menus, language file etc
50-firewall.menu 70-log.menu log.dat en.pl
Requires additional lang files for de, es, fr, it, nl, pl, ru, tr. All others are OK
[v2,6/8] ipblacklist: Ancillary files
sources getipsetstat.c ipblacklistctrl.c
Can be applied without any changes
[v2,8/8] ipblacklist: Build infrastructure
aarch64/stage2 configroot ipblacklist-sources logwatch misc-progs common/stage2 web-user-interface x86_64/stage2 lfs/configroot lfs/ipblacklist-sources lfs/logwatch make.sh misc-progs/Makefile
Missing Files and links:
ipblacklist.conf (for logwatch) ipblacklist (for logwatch) /etc/rc.d/init.d/ipblacklist (Start -up) /etc/rc.d/rcsysinit.d/S87ipblacklist -> ../init.d/ipblacklist
I have written code for the missing files which are working on my firewall and I could submit as patches if required.
/usr/share/logwatch/default.conf/services/ipblacklist.conf /usr/share/logwatch/scripts/services/ipblacklist /etc/rc.d/init.d/ipblacklist
I haven't tried incorporating the patches as I extracted the files from Patchwork and manually inserted them into my firewall for testing. I would be very surprised if there were any problems with Tim's code however.
Hope this will help to progress this very useful addition to IPFire.
Rob
Hi Rob,
In your communication below about the various patches, it sounded to me like you are suggesting that some of the patches could be taken from the earlier patchset because they have not changed. My apologies if I have misread it.
I believe that it would be better to submit a complete patchset now, where you include all patches, even if they haven't changed from earlier. This would make it much easier for tracking the status of the patches within patchwork and knowing what was included at a later date if looking back. So Tim's old patchset would be marked as superseded in patchwork and yours would take over and would be named as a v3 version.
Information on submitting the patches is in the wiki and if needed I can provide the appropriate links. If you have any questions on or problems with the patch submission process then just let me know.
Regards,
Adolf.
On 07/02/2022 13:58, Rob Brewer wrote:
Hi Michael,
Michael Tremer wrote:
Hello Rob,
Thank you for your interest in working on this.
Yes, I always thought that there was great interest in moving this over the line. However, I could not find where we left off here.
There were a couple of outstanding issues that had to be resolved. I just couldn’t find my last emails. Are you aware of these?
You expect have seen the topic :
https://community.ipfire.org/t/can-i-help-progress-version2-of-ipblacklist-a...
where Adolf has been encouraging me to look at this feature and I now have a working copy on my firewall.
I believe the patches on patchwork are mostly complete and with a few additions/corrections can be incorporated into the latest build.
Here is my understanding of where we are:
https://patchwork.ipfire.org/project/ipfire/patch/20200427143123.6378-9-ipfr...
[v2,0/8] ipblacklist: IP Address Blacklists
This patch was applied to convert V1 into V2 and used to make the the remaining patches It can safely be ignored.
[v2,1/8] ipblacklist: Main script This is the ipblacklist main perl program. It can be incorporated without problems.
[v2,2/8] ipblacklist: WUI Settings page ipblacklist.cgi Patch can be applied without any changes.
[v2,3/8] ipblacklist: WUI Log page ipblacklists.dat - Needs to be modified with an additional patch to:
- remove - require “${General::swroot}/geoip-functions.pl”; - This isn’t
included in core 161 and not used in this program.
- change &General::readhash( “/srv/web/ipfire/html/themes/”.
$mainsettings{‘THEME’}."/include/colors.txt", %color ); to &General::readhash( “/srv/web/ipfire/html/themes/ipfire/include/colors.txt”, %color );
[v2,4/8] ipblacklist: WUI Log details page
showrequestfromblacklist.dat - Patch can be applied without any changes
[v2,5/8] ipblacklist: WUI menus, language file etc
50-firewall.menu 70-log.menu log.dat en.pl
Requires additional lang files for de, es, fr, it, nl, pl, ru, tr. All others are OK
[v2,6/8] ipblacklist: Ancillary files
sources getipsetstat.c ipblacklistctrl.c
Can be applied without any changes
[v2,8/8] ipblacklist: Build infrastructure
aarch64/stage2 configroot ipblacklist-sources logwatch misc-progs common/stage2 web-user-interface x86_64/stage2 lfs/configroot lfs/ipblacklist-sources lfs/logwatch make.sh misc-progs/Makefile
Missing Files and links:
ipblacklist.conf (for logwatch) ipblacklist (for logwatch) /etc/rc.d/init.d/ipblacklist (Start -up) /etc/rc.d/rcsysinit.d/S87ipblacklist -> ../init.d/ipblacklist
I have written code for the missing files which are working on my firewall and I could submit as patches if required.
/usr/share/logwatch/default.conf/services/ipblacklist.conf /usr/share/logwatch/scripts/services/ipblacklist /etc/rc.d/init.d/ipblacklist
I haven't tried incorporating the patches as I extracted the files from Patchwork and manually inserted them into my firewall for testing. I would be very surprised if there were any problems with Tim's code however.
Hope this will help to progress this very useful addition to IPFire.
Rob
Hi Adolf,
Thank you for the kind comments in your earlier post and I'm pleased that ipblacklist will be incorporated into IPFire. From what I'm seeing here it will be a very worthwhile addition. I have been looking through the Wiki documentation for submitting patches and it looks fairly explanatory but will probably need your help to keep me in the right place.
Adolf Belka wrote:
Hi Rob,
In your communication below about the various patches, it sounded to me like you are suggesting that some of the patches could be taken from the earlier patchset because they have not changed. My apologies if I have misread it.
I think that is correct. The majority of Tim's V2 patches can be incorporated without changes.
I believe that it would be better to submit a complete patchset now, where you include all patches, even if they haven't changed from earlier. This would make it much easier for tracking the status of the patches within patchwork and knowing what was included at a later date if looking back. So Tim's old patchset would be marked as superseded in patchwork and yours would take over and would be named as a v3 version.
That's OK but it'll get me a couple of days to get my head around submitting patches and getting the correct build environment set up. I presume you would need patches against the next development load (git checkout -b next -t origin/next) which I will have to set up.
Information on submitting the patches is in the wiki and if needed I can provide the appropriate links. If you have any questions on or problems with the patch submission process then just let me know.
Thanks Adolf, I've read some of the docs. It mostly looks OK but when I get around to submitting patches I think I'll need some help on submitting the Build infrastructure patches and I don't have a clue how to generate the lang files.
It looks like the first step will be difficult but then hopefully gets easier.
Regards
Rob
Hi Rob,
On 09/02/2022 22:43, Rob Brewer wrote:
Hi Adolf,
Thank you for the kind comments in your earlier post and I'm pleased that ipblacklist will be incorporated into IPFire. From what I'm seeing here it will be a very worthwhile addition. I have been looking through the Wiki documentation for submitting patches and it looks fairly explanatory but will probably need your help to keep me in the right place.
Just ask when you have any questions.
Adolf Belka wrote:
Hi Rob,
In your communication below about the various patches, it sounded to me like you are suggesting that some of the patches could be taken from the earlier patchset because they have not changed. My apologies if I have misread it.
I think that is correct. The majority of Tim's V2 patches can be incorporated without changes.
I believe that it would be better to submit a complete patchset now, where you include all patches, even if they haven't changed from earlier. This would make it much easier for tracking the status of the patches within patchwork and knowing what was included at a later date if looking back. So Tim's old patchset would be marked as superseded in patchwork and yours would take over and would be named as a v3 version.
That's OK but it'll get me a couple of days to get my head around submitting patches and getting the correct build environment set up. I presume you would need patches against the next development load (git checkout -b next -t origin/next) which I will have to set up.
That is correct.
Information on submitting the patches is in the wiki and if needed I can provide the appropriate links. If you have any questions on or problems with the patch submission process then just let me know.
Thanks Adolf, I've read some of the docs. It mostly looks OK but when I get around to submitting patches I think I'll need some help on submitting the Build infrastructure patches and I don't have a clue how to generate the lang files.
For the languages that are not provided then the default in IPFire is that the English version will be used. This is also the case if some but not all entries are translated. The untranslated entries stay as English.If there are languages where there are no files then you don't have to worry about them. IPFire users who are native speakers usually over time provide the translations, either as patches or as files that one of us can create patches from.
I will send you tomorrow the commands you need to run for the language files after the build command has been completed.
It looks like the first step will be difficult but then hopefully gets easier.
Regards
Rob
Hello,
On 9 Feb 2022, at 22:14, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Rob,
On 09/02/2022 22:43, Rob Brewer wrote:
Hi Adolf,
Thank you for the kind comments in your earlier post and I'm pleased that ipblacklist will be incorporated into IPFire. From what I'm seeing here it will be a very worthwhile addition. I have been looking through the Wiki documentation for submitting patches and it looks fairly explanatory but will probably need your help to keep me in the right place.
Just ask when you have any questions.
Adolf Belka wrote:
Hi Rob,
In your communication below about the various patches, it sounded to me like you are suggesting that some of the patches could be taken from the earlier patchset because they have not changed. My apologies if I have misread it.
I think that is correct. The majority of Tim's V2 patches can be incorporated without changes.
I believe that it would be better to submit a complete patchset now, where you include all patches, even if they haven't changed from earlier. This would make it much easier for tracking the status of the patches within patchwork and knowing what was included at a later date if looking back. So Tim's old patchset would be marked as superseded in patchwork and yours would take over and would be named as a v3 version.
That's OK but it'll get me a couple of days to get my head around submitting patches and getting the correct build environment set up. I presume you would need patches against the next development load (git checkout -b next -t origin/next) which I will have to set up.
That is correct.
Information on submitting the patches is in the wiki and if needed I can provide the appropriate links. If you have any questions on or problems with the patch submission process then just let me know.
Thanks Adolf, I've read some of the docs. It mostly looks OK but when I get around to submitting patches I think I'll need some help on submitting the Build infrastructure patches and I don't have a clue how to generate the lang files.
For the languages that are not provided then the default in IPFire is that the English version will be used. This is also the case if some but not all entries are translated. The untranslated entries stay as English.If there are languages where there are no files then you don't have to worry about them. IPFire users who are native speakers usually over time provide the translations, either as patches or as files that one of us can create patches from.
Let’s not worry about any other language than English for now. Everyone on the team can handle this during the development process and we should translate everything in the end. Otherwise we might change a string or add another checkbox somewhere and we need to reach out to the native speakers of the other languages for just one sentence.
We have lots of native German speakers on here, so that won’t be a problem. Stéphane is a native French speaker who regularly updates the French translation and to whom we should reach out to when the time is right :) Otherwise the language support in IPFire is pretty much best-effort only.
-Michael
I will send you tomorrow the commands you need to run for the language files after the build command has been completed.
It looks like the first step will be difficult but then hopefully gets easier.
Regards
Rob
Hi Michael
Michael Tremer wrote:
Hello Rob,
Thank you for your interest in working on this.
Yes, I always thought that there was great interest in moving this over the line. However, I could not find where we left off here.
There were a couple of outstanding issues that had to be resolved. I just couldn’t find my last emails. Are you aware of these?
I'm pleased to say I have had an email from Tim and is supportive of my attempts to progress ipblacklist into IPFire. Tim however says "Between COVID, my taking on additional responsibilities and the code not being part of ipfire, it's currently got a very low priority for me."
Tim pointed me to his git pages where I was able to find most of the code that I thought was missing from patchwork and is all now installed on my firewall and is working extremely well.
You may be interested in one of the modification I have made to ipblacklist, is to add an additional local blacklist to the sources file to get a blocklist from a web server on my local network. This is populated by a script which greps the mail server logs for SMTP Auth attacks and has been particularly useful in protecting the mail server from a recent botnet attack where the offending ip addresses have been recycled every one to three weeks. Currently the blocklist contains about 3000 ip addresses and has blocked nearly 2000 smtp auth attempts so far to-day.
I also use fail2ban and Banish to manage iptables blocks on the firewall.
The last communication I could find between yourself and Tim was in May 2020. https://lists.ipfire.org/pipermail/development/2020-May/007822.html
Hope this is useful.
Rob
Hi Rob, Good to hear that you had some contact with Tim and he is supportive of you picking this up.
Just to let you know that this ipblacklist was discussed at the monthly IPFire Development Conf call last Monday and your work on this was appreciated. The decision was that it should be a core part of IPFire and not an addon. The view is that IPFire as a firewall should have everything required for a firewall available as default. The only things that should be addons are those that are not a core part of a firewall. An ipblacklist is something that should be available to the firewall as standard. It was also agreed that I should help support you in whatever way I can for the patch submission etc, if required. I will separately reply to your earlier e-mail about the various patches. Michael can give any comment he wants with regard to the earlier communications.
Regards,
Adolf.
On 09/02/2022 14:23, Rob Brewer wrote:
Hi Michael
Michael Tremer wrote:
Hello Rob,
Thank you for your interest in working on this.
Yes, I always thought that there was great interest in moving this over the line. However, I could not find where we left off here.
There were a couple of outstanding issues that had to be resolved. I just couldn’t find my last emails. Are you aware of these?
I'm pleased to say I have had an email from Tim and is supportive of my attempts to progress ipblacklist into IPFire. Tim however says "Between COVID, my taking on additional responsibilities and the code not being part of ipfire, it's currently got a very low priority for me."
Tim pointed me to his git pages where I was able to find most of the code that I thought was missing from patchwork and is all now installed on my firewall and is working extremely well.
You may be interested in one of the modification I have made to ipblacklist, is to add an additional local blacklist to the sources file to get a blocklist from a web server on my local network. This is populated by a script which greps the mail server logs for SMTP Auth attacks and has been particularly useful in protecting the mail server from a recent botnet attack where the offending ip addresses have been recycled every one to three weeks. Currently the blocklist contains about 3000 ip addresses and has blocked nearly 2000 smtp auth attempts so far to-day.
I also use fail2ban and Banish to manage iptables blocks on the firewall.
The last communication I could find between yourself and Tim was in May 2020. https://lists.ipfire.org/pipermail/development/2020-May/007822.html
Hope this is useful.
Rob
Hello Rob,
On 9 Feb 2022, at 13:23, Rob Brewer ipfire-devel@grantura.co.uk wrote:
Hi Michael
Michael Tremer wrote:
Hello Rob,
Thank you for your interest in working on this.
Yes, I always thought that there was great interest in moving this over the line. However, I could not find where we left off here.
There were a couple of outstanding issues that had to be resolved. I just couldn’t find my last emails. Are you aware of these?
I'm pleased to say I have had an email from Tim and is supportive of my attempts to progress ipblacklist into IPFire. Tim however says "Between COVID, my taking on additional responsibilities and the code not being part of ipfire, it's currently got a very low priority for me."
Good to hear that you are in touch. I would like to invite Tim to join the conversation on here. I am sure he has a couple of thoughts to contribute and I hope he can find the time.
Tim pointed me to his git pages where I was able to find most of the code that I thought was missing from patchwork and is all now installed on my firewall and is working extremely well.
I assume you are talking about this here?
https://git.ipfire.org/?p=people/timf/ipfire-2.x.git;a=shortlog;h=refs/heads...
That would have been one of my first questions having looked at my emails again: Get the code into some Git repository.
This is a large patchset and it is very difficult to scroll up and down to review it. Uploading it to a Git repository that is browsable in a web browser somewhere would be a lot better and we can put any patches on top of the branch, so that we only will have smaller changes to review and not a whole patchset again and again.
Do you have a Git repository somewhere? Would you like me to set up your IPFire account so that you can use our servers?
Do you have experience with Git?
We would need to rebase the branch onto next (which Adolf has already pointed out), but I don’t think this would be a problem because we are mainly adding new code and don’t modify too much existing stuff here.
You may be interested in one of the modification I have made to ipblacklist, is to add an additional local blacklist to the sources file to get a blocklist from a web server on my local network. This is populated by a script which greps the mail server logs for SMTP Auth attacks and has been particularly useful in protecting the mail server from a recent botnet attack where the offending ip addresses have been recycled every one to three weeks. Currently the blocklist contains about 3000 ip addresses and has blocked nearly 2000 smtp auth attempts so far to-day.
I also use fail2ban and Banish to manage iptables blocks on the firewall.
This is kind of a fail2ban but on the firewall. Since this patchset is already so large and there has been a custom blocklist existing before which got removed, I would push this project back a little bit until we have a base that we can add new features to.
I am not entirely convinced that this functionality scales well across all users. How would they move the logs to the firewall? We don’t have a simple API, but if we did, we would not have a system to authenticated other servers. This would be a project that I would find a little bit more complicated and we would need a couple more pieces in the puzzle before we are ready for it.
The last communication I could find between yourself and Tim was in May 2020. https://lists.ipfire.org/pipermail/development/2020-May/007822.html
Thanks for finding this. Indeed the conversation just ended in nothingness due to lack of time of everybody I would suspect.
I could not find anything on the list that I would consider a blocker. There are however some smaller things like translations and probably there will be a couple of minor bugs and some improvements to the look and feel.
So, can we start with rebasing the Git branch, please?
-Michael
Hope this is useful.
Rob
Hi Michael
On Thursday 10 February 2022 09:41 Michael Tremer wrote:
Hello Rob,
Good to hear that you are in touch. I would like to invite Tim to join the conversation on here. I am sure he has a couple of thoughts to contribute and I hope he can find the time.
I assume you are talking about this here?
https://git.ipfire.org/?p=people/timf/ipfire-2.x.git;a=shortlog;h=refs/heads...
Yes Tim's link to his ipfire git pages which links your URL.
That would have been one of my first questions having looked at my emails again: Get the code into some Git repository.
This is a large patchset and it is very difficult to scroll up and down to review it. Uploading it to a Git repository that is browsable in a web browser somewhere would be a lot better and we can put any patches on top of the branch, so that we only will have smaller changes to review and not a whole patchset again and again.
I think the very large patch which gave you problems in May 2020 and linked to below and (PATCH v2 0/8 on Patchwork), patched the abandoned version 1 of ipblacklist to version 2. The resulting V2 patches:
ipblacklist: Build infrastructure ipblacklist: Modifications to system ipblacklist: Ancillary files ipblacklist: WUI menus, language file etc ipblacklist: WUI Log details page ipblacklist: WUI Log page ipblacklist: WUI Settings page ipblacklist: Main script
are on: https://git.ipfire.org/?p=people/timf/ipfire-2.x.git;a=shortlog;h=refs/heads...
I think these were (at May 2020) pretty much ready to go and would only need updating if the core files have changed in the meantime. To date the only required patch I am aware of would be to ipblacklists.dat to reflect the changes in the latest version of IPFire. (There could be others as I only patched my system files rather than replaced them)
Do you have a Git repository somewhere? Would you like me to set up your IPFire account so that you can use our servers?
I have a Git Development Environment on this computer (Debian Sid 5.16.0-1- amd64 16GB memory) currently at core163 but is a bit short of room as the disk is showing 95% full. Your offer of an account on IPFire might be a sensible alternative.
Do you have experience with Git?
We would need to rebase the branch onto next (which Adolf has already pointed out), but I don’t think this would be a problem because we are mainly adding new code and don’t modify too much existing stuff here.
I don't think that would be a problem.
You may be interested in one of the modification I have made to ipblacklist, is to add an additional local blacklist to the sources file to get a blocklist from a web server on my local network. This is populated by a script which greps the mail server logs for SMTP Auth attacks and has been particularly useful in protecting the mail server from a recent botnet attack where the offending ip addresses have been recycled every one to three weeks. Currently the blocklist contains about 3000 ip addresses and has blocked nearly 2000 smtp auth attempts so far to-day.
I also use fail2ban and Banish to manage iptables blocks on the firewall.
This is kind of a fail2ban but on the firewall. Since this patchset is already so large and there has been a custom blocklist existing before which got removed, I would push this project back a little bit until we have a base that we can add new features to.
I wasn't suggesting that we add this to the current ipblacklist version. However it only requires 1 extra resource added to the sources file so the modifications are minimal but would have limited user appeal and it does require some local processing of server lo files. However for my botnet attack it is being very effective.
I am not entirely convinced that this functionality scales well across all users. How would they move the logs to the firewall? We don’t have a simple API, but if we did, we would not have a system to authenticated other servers. This would be a project that I would find a little bit more complicated and we would need a couple more pieces in the puzzle before we are ready for it.
I fully agree.
The last communication I could find between yourself and Tim was in May 2020. https://lists.ipfire.org/pipermail/development/2020-May/007822.html
Thanks for finding this. Indeed the conversation just ended in nothingness due to lack of time of everybody I would suspect.
I could not find anything on the list that I would consider a blocker. There are however some smaller things like translations and probably there will be a couple of minor bugs and some improvements to the look and feel.
There are a couple of points we need to consider:
1) IPBlacklist does not work very well if Tim's ipfblocklist add-on is also installed. My view is that the add-on should be removed before IPBlacklist can be applied. Can the add-on be automatically removed on installaion and should we transfer the settings info from ipfbocklist to ipblacklist?
2) I added a init script to my firewall which doesn't seem to be present on Tim's patches. I'm not sure if this is needed as it will be started by fcron or changes made in the WUI but won't be instantly available on re-boot. Do you have any thoughts on this?
So, can we start with rebasing the Git branch, please?
No problem.
Rob
Hello Rob,
On 10 Feb 2022, at 15:12, Rob Brewer ipfire-devel@grantura.co.uk wrote:
Hi Michael
On Thursday 10 February 2022 09:41 Michael Tremer wrote:
Hello Rob,
Good to hear that you are in touch. I would like to invite Tim to join the conversation on here. I am sure he has a couple of thoughts to contribute and I hope he can find the time.
I assume you are talking about this here?
https://git.ipfire.org/?p=people/timf/ipfire-2.x.git;a=shortlog;h=refs/heads...
Yes Tim's link to his ipfire git pages which links your URL.
Okay, I just set up your very own repository over here:
https://git.ipfire.org/?p=people/helix/ipfire-2.x.git;a=summary
I will send you a private email with your account stuff.
That would have been one of my first questions having looked at my emails again: Get the code into some Git repository.
This is a large patchset and it is very difficult to scroll up and down to review it. Uploading it to a Git repository that is browsable in a web browser somewhere would be a lot better and we can put any patches on top of the branch, so that we only will have smaller changes to review and not a whole patchset again and again.
I think the very large patch which gave you problems in May 2020 and linked to below and (PATCH v2 0/8 on Patchwork), patched the abandoned version 1 of ipblacklist to version 2. The resulting V2 patches:
ipblacklist: Build infrastructure ipblacklist: Modifications to system ipblacklist: Ancillary files ipblacklist: WUI menus, language file etc ipblacklist: WUI Log details page ipblacklist: WUI Log page ipblacklist: WUI Settings page ipblacklist: Main script
are on: https://git.ipfire.org/?p=people/timf/ipfire-2.x.git;a=shortlog;h=refs/heads...
I think these were (at May 2020) pretty much ready to go and would only need updating if the core files have changed in the meantime. To date the only required patch I am aware of would be to ipblacklists.dat to reflect the changes in the latest version of IPFire. (There could be others as I only patched my system files rather than replaced them)
What changes are those?
Do you have a Git repository somewhere? Would you like me to set up your IPFire account so that you can use our servers?
I have a Git Development Environment on this computer (Debian Sid 5.16.0-1- amd64 16GB memory) currently at core163 but is a bit short of room as the disk is showing 95% full. Your offer of an account on IPFire might be a sensible alternative.
See above.
Do you have experience with Git?
We would need to rebase the branch onto next (which Adolf has already pointed out), but I don’t think this would be a problem because we are mainly adding new code and don’t modify too much existing stuff here.
I don't think that would be a problem.
Great!
You may be interested in one of the modification I have made to ipblacklist, is to add an additional local blacklist to the sources file to get a blocklist from a web server on my local network. This is populated by a script which greps the mail server logs for SMTP Auth attacks and has been particularly useful in protecting the mail server from a recent botnet attack where the offending ip addresses have been recycled every one to three weeks. Currently the blocklist contains about 3000 ip addresses and has blocked nearly 2000 smtp auth attempts so far to-day.
I also use fail2ban and Banish to manage iptables blocks on the firewall.
This is kind of a fail2ban but on the firewall. Since this patchset is already so large and there has been a custom blocklist existing before which got removed, I would push this project back a little bit until we have a base that we can add new features to.
I wasn't suggesting that we add this to the current ipblacklist version. However it only requires 1 extra resource added to the sources file so the modifications are minimal but would have limited user appeal and it does require some local processing of server lo files. However for my botnet attack it is being very effective.
Okay. It is great to see that this works wonders already :)
I am not entirely convinced that this functionality scales well across all users. How would they move the logs to the firewall? We don’t have a simple API, but if we did, we would not have a system to authenticated other servers. This would be a project that I would find a little bit more complicated and we would need a couple more pieces in the puzzle before we are ready for it.
I fully agree.
The last communication I could find between yourself and Tim was in May 2020. https://lists.ipfire.org/pipermail/development/2020-May/007822.html
Thanks for finding this. Indeed the conversation just ended in nothingness due to lack of time of everybody I would suspect.
I could not find anything on the list that I would consider a blocker. There are however some smaller things like translations and probably there will be a couple of minor bugs and some improvements to the look and feel.
There are a couple of points we need to consider:
- IPBlacklist does not work very well if Tim's ipfblocklist add-on is also
installed. My view is that the add-on should be removed before IPBlacklist can be applied. Can the add-on be automatically removed on installaion and should we transfer the settings info from ipfbocklist to ipblacklist?
Yes, in theory we could remove any old files in the updater and install our own ones.
- I added a init script to my firewall which doesn't seem to be present on
Tim's patches. I'm not sure if this is needed as it will be started by fcron or changes made in the WUI but won't be instantly available on re-boot. Do you have any thoughts on this?
What does the script do?
So, can we start with rebasing the Git branch, please?
No problem.
Rob
-Michael
Hello,
On 10/02/2022 16:48, Michael Tremer wrote:
Hello Rob,
On 10 Feb 2022, at 15:12, Rob Brewer ipfire-devel@grantura.co.uk wrote:
Hi Michael
On Thursday 10 February 2022 09:41 Michael Tremer wrote:
Hello Rob,
Good to hear that you are in touch. I would like to invite Tim to join the conversation on here. I am sure he has a couple of thoughts to contribute and I hope he can find the time.
I assume you are talking about this here?
https://git.ipfire.org/?p=people/timf/ipfire-2.x.git;a=shortlog;h=refs/heads...
Yes Tim's link to his ipfire git pages which links your URL.
Okay, I just set up your very own repository over here:
https://git.ipfire.org/?p=people/helix/ipfire-2.x.git;a=summary
I will send you a private email with your account stuff.
That would have been one of my first questions having looked at my emails again: Get the code into some Git repository.
This is a large patchset and it is very difficult to scroll up and down to review it. Uploading it to a Git repository that is browsable in a web browser somewhere would be a lot better and we can put any patches on top of the branch, so that we only will have smaller changes to review and not a whole patchset again and again.
I think the very large patch which gave you problems in May 2020 and linked to below and (PATCH v2 0/8 on Patchwork), patched the abandoned version 1 of ipblacklist to version 2. The resulting V2 patches:
ipblacklist: Build infrastructure ipblacklist: Modifications to system ipblacklist: Ancillary files ipblacklist: WUI menus, language file etc ipblacklist: WUI Log details page ipblacklist: WUI Log page ipblacklist: WUI Settings page ipblacklist: Main script
are on: https://git.ipfire.org/?p=people/timf/ipfire-2.x.git;a=shortlog;h=refs/heads...
I think these were (at May 2020) pretty much ready to go and would only need updating if the core files have changed in the meantime. To date the only required patch I am aware of would be to ipblacklists.dat to reflect the changes in the latest version of IPFire. (There could be others as I only patched my system files rather than replaced them)
What changes are those?
Do you have a Git repository somewhere? Would you like me to set up your IPFire account so that you can use our servers?
I have a Git Development Environment on this computer (Debian Sid 5.16.0-1- amd64 16GB memory) currently at core163 but is a bit short of room as the disk is showing 95% full. Your offer of an account on IPFire might be a sensible alternative.
See above.
Do you have experience with Git?
We would need to rebase the branch onto next (which Adolf has already pointed out), but I don’t think this would be a problem because we are mainly adding new code and don’t modify too much existing stuff here.
I don't think that would be a problem.
Great!
You may be interested in one of the modification I have made to ipblacklist, is to add an additional local blacklist to the sources file to get a blocklist from a web server on my local network. This is populated by a script which greps the mail server logs for SMTP Auth attacks and has been particularly useful in protecting the mail server from a recent botnet attack where the offending ip addresses have been recycled every one to three weeks. Currently the blocklist contains about 3000 ip addresses and has blocked nearly 2000 smtp auth attempts so far to-day.
I also use fail2ban and Banish to manage iptables blocks on the firewall.
This is kind of a fail2ban but on the firewall. Since this patchset is already so large and there has been a custom blocklist existing before which got removed, I would push this project back a little bit until we have a base that we can add new features to.
I wasn't suggesting that we add this to the current ipblacklist version. However it only requires 1 extra resource added to the sources file so the modifications are minimal but would have limited user appeal and it does require some local processing of server lo files. However for my botnet attack it is being very effective.
Okay. It is great to see that this works wonders already :)
I am not entirely convinced that this functionality scales well across all users. How would they move the logs to the firewall? We don’t have a simple API, but if we did, we would not have a system to authenticated other servers. This would be a project that I would find a little bit more complicated and we would need a couple more pieces in the puzzle before we are ready for it.
I fully agree.
This sounds as if it does the same sort of thing as something I had in my first patchset. I added an extra rule to the input policy chain that added the address to an ipset if the number of dropped packets exceeded a threshold. This runs completely within iptables/ipset.
iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist src -j SET --add-set $autoblacklist src --exist" ); iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist dst -j SET --add-set $autoblacklist dst --exist" ); iptables( "-I POLICYIN 1 -i $red_iface -m hashlimit --hashlimit-mode srcip --hashlimit-above $settings{BLOCK_THRESHOLD}/hour --hashlimit-name $autoblacklist -j SET --add-set $autoblacklist src" );
The last communication I could find between yourself and Tim was in May 2020. https://lists.ipfire.org/pipermail/development/2020-May/007822.html
Thanks for finding this. Indeed the conversation just ended in nothingness due to lack of time of everybody I would suspect.
I could not find anything on the list that I would consider a blocker. There are however some smaller things like translations and probably there will be a couple of minor bugs and some improvements to the look and feel.
There are a couple of points we need to consider:
- IPBlacklist does not work very well if Tim's ipfblocklist add-on is also
installed. My view is that the add-on should be removed before IPBlacklist can be applied. Can the add-on be automatically removed on installaion and should we transfer the settings info from ipfbocklist to ipblacklist?
Yes, in theory we could remove any old files in the updater and install our own ones.
- I added a init script to my firewall which doesn't seem to be present on
Tim's patches. I'm not sure if this is needed as it will be started by fcron or changes made in the WUI but won't be instantly available on re-boot. Do you have any thoughts on this?
I don't think this is needed - the change to the firewall init script should call the ipblacklist script at the correct time.
What does the script do?
So, can we start with rebasing the Git branch, please?
No problem.
Rob
-Michael
Tim
Hi Tim, Good to see your posing here.
On Saturday 12 February 2022 21:29 Tim FitzGeorge wrote:
This sounds as if it does the same sort of thing as something I had in my first patchset. I added an extra rule to the input policy chain that added the address to an ipset if the number of dropped packets exceeded a
threshold. This runs completely within iptables/ipset.
iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist src -j SET --add-set $autoblacklist src --exist" ); iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist dst -j SET --add-set $autoblacklist dst --exist" ); iptables( "-I POLICYIN 1 -i $red_iface -m hashlimit --hashlimit-mode srcip --hashlimit-above $settings{BLOCK_THRESHOLD}/hour --hashlimit-name $autoblacklist -j SET --add-set $autoblacklist src" );
Aotoblacklist looks like a useful addition. Why did you drop t?
There are a couple of points we need to consider:
- IPBlacklist does not work very well if Tim's ipfblocklist add-on is also
installed. My view is that the add-on should be removed before IPBlacklist can be applied. Can the add-on be automatically removed on installaion and should we transfer the settings info from ipfbocklist to ipblacklist?
Yes, in theory we could remove any old files in the updater and install our own ones.
There are a couple of errors on your uninstall-blocklist.sh script which leaves some files behind when it is run. I can send you a patch for this if it is of help.
- I added a init script to my firewall which doesn't seem to be present on
Tim's patches. I'm not sure if this is needed as it will be started by fcron or changes made in the WUI but won't be instantly available on re-boot. Do you have any thoughts on this?
I don't think this is needed - the change to the firewall init script should call the ipblacklist script at the correct time.
I hadn't noticed the last few lines in your firewall init script which my init script duplicates. So I agree my addition isn't needed.
I have started producing the v3 patches requested by the devs, but apart from the couple of changes needed to ipblacklists.dat I think they will be almost identical to your v2 patches.
Rob
Hello,
On 13 Feb 2022, at 12:44, Rob Brewer ipfire-devel@grantura.co.uk wrote:
Hi Tim, Good to see your posing here.
On Saturday 12 February 2022 21:29 Tim FitzGeorge wrote:
This sounds as if it does the same sort of thing as something I had in my first patchset. I added an extra rule to the input policy chain that added the address to an ipset if the number of dropped packets exceeded a
threshold. This runs completely within iptables/ipset.
iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist src -j SET --add-set $autoblacklist src --exist" ); iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist dst -j SET --add-set $autoblacklist dst --exist" ); iptables( "-I POLICYIN 1 -i $red_iface -m hashlimit --hashlimit-mode srcip --hashlimit-above $settings{BLOCK_THRESHOLD}/hour --hashlimit-name $autoblacklist -j SET --add-set $autoblacklist src" );
Aotoblacklist looks like a useful addition. Why did you drop t?
I asked for that. I believed that it would causing more trouble than it helps.
The default policy of the firewall is dropping traffic anyways. Adding an extra rule does not change behaviour of the firewall and I wasn’t sure if it would add some false sense of extra security.
Since the firewall has no idea who is hammering some web application trying out passwords I did not see the value if that makes sense.
It would be great to have some more insight into who the big offenders are, but I don’t believe that this is the right place.
There are a couple of points we need to consider:
- IPBlacklist does not work very well if Tim's ipfblocklist add-on is also
installed. My view is that the add-on should be removed before IPBlacklist can be applied. Can the add-on be automatically removed on installaion and should we transfer the settings info from ipfbocklist to ipblacklist?
Yes, in theory we could remove any old files in the updater and install our own ones.
There are a couple of errors on your uninstall-blocklist.sh script which leaves some files behind when it is run. I can send you a patch for this if it is of help.
Since this is going into the main distribution we won’t need this script any more, but you can make it work for testing if you like.
- I added a init script to my firewall which doesn't seem to be present on
Tim's patches. I'm not sure if this is needed as it will be started by fcron or changes made in the WUI but won't be instantly available on re-boot. Do you have any thoughts on this?
I don't think this is needed - the change to the firewall init script should call the ipblacklist script at the correct time.
I hadn't noticed the last few lines in your firewall init script which my init script duplicates. So I agree my addition isn't needed.
I have started producing the v3 patches requested by the devs, but apart from the couple of changes needed to ipblacklists.dat I think they will be almost identical to your v2 patches.
It isn’t bad when things are identical. Good code is good code and doesn’t need to be changed just for the sake of it.
Your Git repository is still empty, do you know how to push anything into it? I am mostly curious if I set it up okay that it will work :)
-Michael
Rob
Hello Tim,
Welcome back :)
On 12 Feb 2022, at 21:29, Tim FitzGeorge ipfr@tfitzgeorge.me.uk wrote:
Hello,
On 10/02/2022 16:48, Michael Tremer wrote:
Hello Rob,
On 10 Feb 2022, at 15:12, Rob Brewer ipfire-devel@grantura.co.uk wrote:
Hi Michael
On Thursday 10 February 2022 09:41 Michael Tremer wrote:
Hello Rob,
Good to hear that you are in touch. I would like to invite Tim to join the conversation on here. I am sure he has a couple of thoughts to contribute and I hope he can find the time.
I assume you are talking about this here?
https://git.ipfire.org/?p=people/timf/ipfire-2.x.git;a=shortlog;h=refs/heads...
Yes Tim's link to his ipfire git pages which links your URL.
Okay, I just set up your very own repository over here:
https://git.ipfire.org/?p=people/helix/ipfire-2.x.git;a=summary
I will send you a private email with your account stuff.
That would have been one of my first questions having looked at my emails again: Get the code into some Git repository.
This is a large patchset and it is very difficult to scroll up and down to review it. Uploading it to a Git repository that is browsable in a web browser somewhere would be a lot better and we can put any patches on top of the branch, so that we only will have smaller changes to review and not a whole patchset again and again.
I think the very large patch which gave you problems in May 2020 and linked to below and (PATCH v2 0/8 on Patchwork), patched the abandoned version 1 of ipblacklist to version 2. The resulting V2 patches:
ipblacklist: Build infrastructure ipblacklist: Modifications to system ipblacklist: Ancillary files ipblacklist: WUI menus, language file etc ipblacklist: WUI Log details page ipblacklist: WUI Log page ipblacklist: WUI Settings page ipblacklist: Main script
are on: https://git.ipfire.org/?p=people/timf/ipfire-2.x.git;a=shortlog;h=refs/heads...
I think these were (at May 2020) pretty much ready to go and would only need updating if the core files have changed in the meantime. To date the only required patch I am aware of would be to ipblacklists.dat to reflect the changes in the latest version of IPFire. (There could be others as I only patched my system files rather than replaced them)
What changes are those?
Do you have a Git repository somewhere? Would you like me to set up your IPFire account so that you can use our servers?
I have a Git Development Environment on this computer (Debian Sid 5.16.0-1- amd64 16GB memory) currently at core163 but is a bit short of room as the disk is showing 95% full. Your offer of an account on IPFire might be a sensible alternative.
See above.
Do you have experience with Git?
We would need to rebase the branch onto next (which Adolf has already pointed out), but I don’t think this would be a problem because we are mainly adding new code and don’t modify too much existing stuff here.
I don't think that would be a problem.
Great!
You may be interested in one of the modification I have made to ipblacklist, is to add an additional local blacklist to the sources file to get a blocklist from a web server on my local network. This is populated by a script which greps the mail server logs for SMTP Auth attacks and has been particularly useful in protecting the mail server from a recent botnet attack where the offending ip addresses have been recycled every one to three weeks. Currently the blocklist contains about 3000 ip addresses and has blocked nearly 2000 smtp auth attempts so far to-day.
I also use fail2ban and Banish to manage iptables blocks on the firewall.
This is kind of a fail2ban but on the firewall. Since this patchset is already so large and there has been a custom blocklist existing before which got removed, I would push this project back a little bit until we have a base that we can add new features to.
I wasn't suggesting that we add this to the current ipblacklist version. However it only requires 1 extra resource added to the sources file so the modifications are minimal but would have limited user appeal and it does require some local processing of server lo files. However for my botnet attack it is being very effective.
Okay. It is great to see that this works wonders already :)
I am not entirely convinced that this functionality scales well across all users. How would they move the logs to the firewall? We don’t have a simple API, but if we did, we would not have a system to authenticated other servers. This would be a project that I would find a little bit more complicated and we would need a couple more pieces in the puzzle before we are ready for it.
I fully agree.
This sounds as if it does the same sort of thing as something I had in my first patchset. I added an extra rule to the input policy chain that added the address to an ipset if the number of dropped packets exceeded a threshold. This runs completely within iptables/ipset.
iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist src -j SET --add-set $autoblacklist src --exist" ); iptables( "-I ${autoblacklist}_BLOCK -m set --match-set $autoblacklist dst -j SET --add-set $autoblacklist dst --exist" ); iptables( "-I POLICYIN 1 -i $red_iface -m hashlimit --hashlimit-mode srcip --hashlimit-above $settings{BLOCK_THRESHOLD}/hour --hashlimit-name $autoblacklist -j SET --add-set $autoblacklist src" );
The last communication I could find between yourself and Tim was in May 2020. https://lists.ipfire.org/pipermail/development/2020-May/007822.html
Thanks for finding this. Indeed the conversation just ended in nothingness due to lack of time of everybody I would suspect.
I could not find anything on the list that I would consider a blocker. There are however some smaller things like translations and probably there will be a couple of minor bugs and some improvements to the look and feel.
There are a couple of points we need to consider:
- IPBlacklist does not work very well if Tim's ipfblocklist add-on is also
installed. My view is that the add-on should be removed before IPBlacklist can be applied. Can the add-on be automatically removed on installaion and should we transfer the settings info from ipfbocklist to ipblacklist?
Yes, in theory we could remove any old files in the updater and install our own ones.
- I added a init script to my firewall which doesn't seem to be present on
Tim's patches. I'm not sure if this is needed as it will be started by fcron or changes made in the WUI but won't be instantly available on re-boot. Do you have any thoughts on this?
I don't think this is needed - the change to the firewall init script should call the ipblacklist script at the correct time.
What does the script do?
So, can we start with rebasing the Git branch, please?
No problem.
Rob
-Michael
Tim
Hello Rob, Hello Tim, Hello *,
as anounced on this list, I'm currently working on getting the ipblacklist feature as a core component into IPFire.
I already had a look on the code, which looks nice and very clean to me. As I'm currently also working on getting all ipset related set stuff and rule creation under one hood, this perfectly fits to this.
So my idea to put the ipblacklist feature over the line, was to split some parts of the ipblacklist "main script" (especially the ipset and iptables related stuff) into the perl-based script which is responsible for iptables rule creation.
In this case some other parts of the script (which where necessary in the past, because ipblacklist initial has been designed as an addon) also can be stipped.
Affected parts for example would be the "start", "stop", "enable" and "disable" code, which is not longer required and therefore safely can be dropped.
In the very end the main task for the script would be to download, update, convert and store the blacklists into an ipset compatible format.
Apart from this, I currently do not see any bigger changes for the WUI related stuff.
@Tim: I hope these changes are okay for you.
Getting started, I noticed, that there currently are two git repositories available, which contain the source for ipblacklist.
There is the origin one from Tim and a slightly modified (fixed) v3 version from Rob. I' currently trying to determine, which one would be the best to start from - are there any deeper changes/differences between them?
Please feel free to ask any kind of questions or share you opinion. As usual, I'll share any progress here.
Best regards,
-Stefan
Hi.
I have been looking at Tim FitzGeorge's code for ipblacklist v2 on https://patchwork.ipfire.org/project/ipfire/list/?series=1215%C2%A0to see if I can help progress its incorporation into IPFire. After I extracted the programs from Patchwork I have been able to build them into my firewall where they are running very successfully. The code on the server seems to be in good shape and apart from a few small patches and additions of a few missing scripts I think it could be successfully introduced into the IPFire code base. I am more than happy to help in seeing this process carried out but need to know if this is acceptable to yourselves.
Regards Rob Brewer
Hello Stefan,
Thank you for diving into this.
As you will all have seen, Stefan and Peter have recently migrated things away from xt_geoip and used ipset which is also being used for the blocklists. It would be great to combine all the code so that we have to maintain as little as possible.
On that note, something that I didn’t bring up yet, but which should probably be raised is that “blacklist” is probably not a good name any more in 2022. We use that term in many other places and didn’t change it yet, however, I would like to lead with a good example and add new code only under the more current names. I don’t have the intention to change the existing places myself, but I would be happy to accept a patch just so that we are consistent. I also believe that “blocklist” and “allowlist” describe the whole functionality much better than black and white.
I am not sure whether it is a good idea to rename everything rather sooner or at the end of the development process. Frankly it doesn’t matter, but it would be cool if the patchset put forward for merging into next would have this change.
On top of that I have a number of random things we would need to consider:
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=config/kernel/kernel.confi...
CONFIG_IP_SET_MAX=256
This is the maximum number of ipsets the kernel is able to load at a time. Are we in any danger that we might hit this limit? If some (and yes, they shouldn’t, but the bloody do) enables all countries on the location filter that brings them already to 260. I would say that we should potentially raise this to 1024 or 2048 depending on how many people might make a lot of use of the AS block feature when that arrives.
I also stumbled over the hash size parameter. Stefan reported that the new DROP HOSTILE feature is slowing down his network, which is quite likely happening because of us hardcoding the hash size to 1024 (which is also the default for ipset).
Investigating any further, and testing a little bit around with memory overhead, I rewrite this in libloc:
https://git.ipfire.org/?p=location/libloc.git;a=commitdiff;h=47de14b01a0fae6...
We are now counting the networks that we export (which is a little bit complicated, but doesn’t matter in this context) and then calculate the correct hash size with a factor. Choosing that factor is now the hard part and I have decided to start with 0.75. That means that we will definitely waste some memory, but we will hopefully not have to perform any linked list search whatsoever. Maybe this will need some adjustment, but it looks good to me. See the comment message for more details.
In IP blocklist, we have a factor of one which we potentially might want to look at as well, since those lists will be hit a lot. Has anyone conducted any benchmarks already?
-Michael
On 28 Feb 2022, at 19:32, Stefan Schantl stefan.schantl@ipfire.org wrote:
Hello Rob, Hello Tim, Hello *,
as anounced on this list, I'm currently working on getting the ipblacklist feature as a core component into IPFire.
I already had a look on the code, which looks nice and very clean to me. As I'm currently also working on getting all ipset related set stuff and rule creation under one hood, this perfectly fits to this.
So my idea to put the ipblacklist feature over the line, was to split some parts of the ipblacklist "main script" (especially the ipset and iptables related stuff) into the perl-based script which is responsible for iptables rule creation.
In this case some other parts of the script (which where necessary in the past, because ipblacklist initial has been designed as an addon) also can be stipped.
Affected parts for example would be the "start", "stop", "enable" and "disable" code, which is not longer required and therefore safely can be dropped.
In the very end the main task for the script would be to download, update, convert and store the blacklists into an ipset compatible format.
Apart from this, I currently do not see any bigger changes for the WUI related stuff.
@Tim: I hope these changes are okay for you.
Getting started, I noticed, that there currently are two git repositories available, which contain the source for ipblacklist.
There is the origin one from Tim and a slightly modified (fixed) v3 version from Rob. I' currently trying to determine, which one would be the best to start from - are there any deeper changes/differences between them?
Please feel free to ask any kind of questions or share you opinion. As usual, I'll share any progress here.
Best regards,
-Stefan
Hi.
I have been looking at Tim FitzGeorge's code for ipblacklist v2 on https://patchwork.ipfire.org/project/ipfire/list/?series=1215 to see if I can help progress its incorporation into IPFire. After I extracted the programs from Patchwork I have been able to build them into my firewall where they are running very successfully. The code on the server seems to be in good shape and apart from a few small patches and additions of a few missing scripts I think it could be successfully introduced into the IPFire code base. I am more than happy to help in seeing this process carried out but need to know if this is acceptable to yourselves.
Regards Rob Brewer
Hi Stefan
On Monday 28 February 2022 19:32 Stefan Schantl wrote:
Hello Rob, Hello Tim, Hello *,
as anounced on this list, I'm currently working on getting the ipblacklist feature as a core component into IPFire.
I already had a look on the code, which looks nice and very clean to me. As I'm currently also working on getting all ipset related set stuff and rule creation under one hood, this perfectly fits to this.
I have been in contact with Michael and has kept me up to date with your progress. As I have found out during a recent DOS attack @Tim's ipblacklist is very powerful tool in combating unwanted internet traffic. I had been using @Tim's original ipfblocklist addon which had limitations when trying to use larger blocklists, which @Tim had fixed in his later ipblacklist version.
Using @Tims repositories I was able to recover the code and install ipblacklist on my current firewall (core 161) and confirm that the code was fully functional.
Looking back at the history of ipblacklist the first version contained a dynamic blocklist feature which was dropped in favour of the one now available (V2) but was a patched version of the original V1 code.
My V3 version is a rework of @Tim's V2 code with the introduction of some of the missing unchanged patches from the V1 code plus updated patches to track the core changes from when it was last worked on about 2 years ago.
So my idea to put the ipblacklist feature over the line, was to split some parts of the ipblacklist "main script" (especially the ipset and iptables related stuff) into the perl-based script which is responsible for iptables rule creation.
That seems to be a sensible approach.
In this case some other parts of the script (which where necessary in the past, because ipblacklist initial has been designed as an addon) also can be stipped.
Affected parts for example would be the "start", "stop", "enable" and "disable" code, which is not longer required and therefore safely can be dropped.
I think that is correct. There are several processes that are started and stopped in the firewall script but need to be run but in a controlled order.
In the very end the main task for the script would be to download, update, convert and store the blacklists into an ipset compatible format.
Apart from this, I currently do not see any bigger changes for the WUI related stuff.
@Tim: I hope these changes are okay for you.
Getting started, I noticed, that there currently are two git repositories available, which contain the source for ipblacklist.
There is the origin one from Tim and a slightly modified (fixed) v3 version from Rob. I' currently trying to determine, which one would be the best to start from - are there any deeper changes/differences between them?
I have integrated my V3 patches into core 163 and have a build running on an apu2 new install. I haven't found any problems with this install.
Please feel free to ask any kind of questions or share you opinion. As usual, I'll share any progress here.
Best regards,
-Stefan
Rob
Hello *,
I've made some development progress, which I want to share here:
Most parts of the main backend script ("ipblacklist") from Tim and Rob are ported into a new functions library (ipblocklist-functions.pl) and into the main firewall script (rules.pl).
This process is almost finished and currently allows to create the firewall rules, download the blocklists and to convert them into an ipset compatible format.
Next step will be to import the frontend code (WUI) and adjust it to use the backend code (functions) from the "ipblocklist-functions.pl".
At this time the blocklist feature should be in a use-able state again and I'll go to create an automatic update script and to import all the logging pages stuff etc.
The development progress and single commits can be found here:
https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=shortlog;h=refs/hea...
As usual please feel free to ask any questions or to share your opinion here.
I wish you a nice day,
-Stefan
Hello Rob, Hello Tim, Hello *,
as anounced on this list, I'm currently working on getting the ipblacklist feature as a core component into IPFire.
I already had a look on the code, which looks nice and very clean to me. As I'm currently also working on getting all ipset related set stuff and rule creation under one hood, this perfectly fits to this.
So my idea to put the ipblacklist feature over the line, was to split some parts of the ipblacklist "main script" (especially the ipset and iptables related stuff) into the perl-based script which is responsible for iptables rule creation.
In this case some other parts of the script (which where necessary in the past, because ipblacklist initial has been designed as an addon) also can be stipped.
Affected parts for example would be the "start", "stop", "enable" and "disable" code, which is not longer required and therefore safely can be dropped.
In the very end the main task for the script would be to download, update, convert and store the blacklists into an ipset compatible format.
Apart from this, I currently do not see any bigger changes for the WUI related stuff.
@Tim: I hope these changes are okay for you.
Getting started, I noticed, that there currently are two git repositories available, which contain the source for ipblacklist.
There is the origin one from Tim and a slightly modified (fixed) v3 version from Rob. I' currently trying to determine, which one would be the best to start from - are there any deeper changes/differences between them?
Please feel free to ask any kind of questions or share you opinion. As usual, I'll share any progress here.
Best regards,
-Stefan
Hi.
I have been looking at Tim FitzGeorge's code for ipblacklist v2 on https://patchwork.ipfire.org/project/ipfire/list/?series=1215%C2%A0to see if I can help progress its incorporation into IPFire. After I extracted the programs from Patchwork I have been able to build them into my firewall where they are running very successfully. The code on the server seems to be in good shape and apart from a few small patches and additions of a few missing scripts I think it could be successfully introduced into the IPFire code base. I am more than happy to help in seeing this process carried out but need to know if this is acceptable to yourselves.
Regards Rob Brewer
Hi Stefan
On Saturday 05 March 2022 18:52 Stefan Schantl wrote:
Hello *,
I've made some development progress, which I want to share here:
Most parts of the main backend script ("ipblacklist") from Tim and Rob are ported into a new functions library (ipblocklist-functions.pl) and into the main firewall script (rules.pl).
Good
This process is almost finished and currently allows to create the firewall rules, download the blocklists and to convert them into an ipset compatible format.
Next step will be to import the frontend code (WUI) and adjust it to use the backend code (functions) from the "ipblocklist-functions.pl".
At this time the blocklist feature should be in a use-able state again and I'll go to create an automatic update script and to import all the logging pages stuff etc.
The development progress and single commits can be found here:
https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=shortlog;h=refs/hea...
As usual please feel free to ask any questions or to share your opinion here.
I wish you a nice day,
-Stefan
Great progress. I did find a bug I introduced when I modified the ipblacklist V2 perl script to add a space after the log-prefix BLKLST entry to make the logs compatible with other log-prefixes. This affected showrequestfromblacklist.dat and the modified version 'modified regex for V3 log-prefix added space' should be used.
https://git.ipfire.org/?p=people/helix/ipfire-2.x.git;a=commit;h=2ccc47f1944...
Rob
Hello Rob,
On 5 Mar 2022, at 21:46, Rob Brewer ipfire-devel@grantura.co.uk wrote:
Hi Stefan
On Saturday 05 March 2022 18:52 Stefan Schantl wrote:
Hello *,
I've made some development progress, which I want to share here:
Most parts of the main backend script ("ipblacklist") from Tim and Rob are ported into a new functions library (ipblocklist-functions.pl) and into the main firewall script (rules.pl).
Good
This process is almost finished and currently allows to create the firewall rules, download the blocklists and to convert them into an ipset compatible format.
Next step will be to import the frontend code (WUI) and adjust it to use the backend code (functions) from the "ipblocklist-functions.pl".
At this time the blocklist feature should be in a use-able state again and I'll go to create an automatic update script and to import all the logging pages stuff etc.
The development progress and single commits can be found here:
https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=shortlog;h=refs/hea...
As usual please feel free to ask any questions or to share your opinion here.
I wish you a nice day,
-Stefan
Great progress. I did find a bug I introduced when I modified the ipblacklist V2 perl script to add a space after the log-prefix BLKLST entry to make the logs compatible with other log-prefixes. This affected showrequestfromblacklist.dat and the modified version 'modified regex for V3 log-prefix added space' should be used.
https://git.ipfire.org/?p=people/helix/ipfire-2.x.git;a=commit;h=2ccc47f1944...
Good catch, but wouldn’t it be helpful to add the space to all log prefixes so that it is always easily readable for humans, too?
-Michael
Rob
On Monday 07 March 2022 20:39 Michael Tremer wrote:
Hello Rob,
On 5 Mar 2022, at 21:46, Rob Brewer ipfire-devel@grantura.co.uk wrote:
Hi Stefan
On Saturday 05 March 2022 18:52 Stefan Schantl wrote:
Hello *,
I've made some development progress, which I want to share here:
Most parts of the main backend script ("ipblacklist") from Tim and Rob are ported into a new functions library (ipblocklist-functions.pl) and into the main firewall script (rules.pl).
Good
This process is almost finished and currently allows to create the firewall rules, download the blocklists and to convert them into an ipset compatible format.
Next step will be to import the frontend code (WUI) and adjust it to use the backend code (functions) from the "ipblocklist-functions.pl".
At this time the blocklist feature should be in a use-able state again and I'll go to create an automatic update script and to import all the logging pages stuff etc.
The development progress and single commits can be found here:
https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=shortlog;h=refs/hea...
As usual please feel free to ask any questions or to share your opinion here.
I wish you a nice day,
-Stefan
Great progress. I did find a bug I introduced when I modified the ipblacklist V2 perl script to add a space after the log-prefix BLKLST entry to make the logs compatible with other log-prefixes. This affected showrequestfromblacklist.dat and the modified version 'modified regex for V3 log-prefix added space' should be used.
https://git.ipfire.org/?p=people/helix/ipfire-2.x.git;a=commit;h=2ccc47f1944...
Good catch, but wouldn’t it be helpful to add the space to all log prefixes so that it is always easily readable for humans, too?
-Michael
I think ipblocklist was the only logfile entry to have this problem.
Rob
On Monday 07 March 2022 22:54 Rob Brewer wrote:
On Monday 07 March 2022 20:39 Michael Tremer wrote:
Hello Rob,
On 5 Mar 2022, at 21:46, Rob Brewer ipfire-devel@grantura.co.uk wrote:
Hi Stefan
On Saturday 05 March 2022 18:52 Stefan Schantl wrote:
Hello *,
I've made some development progress, which I want to share here:
Most parts of the main backend script ("ipblacklist") from Tim and Rob are ported into a new functions library (ipblocklist-functions.pl) and into the main firewall script (rules.pl).
Good
This process is almost finished and currently allows to create the firewall rules, download the blocklists and to convert them into an ipset compatible format.
Next step will be to import the frontend code (WUI) and adjust it to use the backend code (functions) from the "ipblocklist-functions.pl".
At this time the blocklist feature should be in a use-able state again and I'll go to create an automatic update script and to import all the logging pages stuff etc.
The development progress and single commits can be found here:
https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=shortlog;h=refs/hea...
As usual please feel free to ask any questions or to share your opinion here.
I wish you a nice day,
-Stefan
Great progress. I did find a bug I introduced when I modified the ipblacklist V2 perl script to add a space after the log-prefix BLKLST entry to make the logs compatible with other log-prefixes. This affected showrequestfromblacklist.dat and the modified version 'modified regex for V3 log-prefix added space' should be used.
https://git.ipfire.org/?p=people/helix/ipfire-2.x.git;a=commit;h=2ccc47f1944...
Good catch, but wouldn’t it be helpful to add the space to all log prefixes so that it is always easily readable for humans, too?
-Michael
I think ipblocklist was the only logfile entry to have this problem.
Rob
I think there is a missing space in rules.pl before tha last "" I think line 755 should be:
run("$IPTABLES -A ${blocklist}_DROP -j LOG -m limit --limit 10/second --log-prefix "BLKLST_$blocklist "");
and in my ipblocklists.dat I changed your line 298 to be compatible with core 163 as I don't think you can change the 'theme' now and produced an error.
from: &General::readhash("/srv/web/ipfire/html/themes/".$mainsettings{'THEME'}."/include/colors.txt", %color);
to: &General::readhash("/srv/web/ipfire/html/themes/ipfire/include/colors.txt", %color);
Rob
Hello Rob,
On 8 Mar 2022, at 10:59, Rob Brewer ipfire-devel@grantura.co.uk wrote:
On Monday 07 March 2022 22:54 Rob Brewer wrote:
On Monday 07 March 2022 20:39 Michael Tremer wrote:
Hello Rob,
On 5 Mar 2022, at 21:46, Rob Brewer ipfire-devel@grantura.co.uk wrote:
Hi Stefan
On Saturday 05 March 2022 18:52 Stefan Schantl wrote:
Hello *,
I've made some development progress, which I want to share here:
Most parts of the main backend script ("ipblacklist") from Tim and Rob are ported into a new functions library (ipblocklist-functions.pl) and into the main firewall script (rules.pl).
Good
This process is almost finished and currently allows to create the firewall rules, download the blocklists and to convert them into an ipset compatible format.
Next step will be to import the frontend code (WUI) and adjust it to use the backend code (functions) from the "ipblocklist-functions.pl".
At this time the blocklist feature should be in a use-able state again and I'll go to create an automatic update script and to import all the logging pages stuff etc.
The development progress and single commits can be found here:
https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=shortlog;h=refs/hea...
As usual please feel free to ask any questions or to share your opinion here.
I wish you a nice day,
-Stefan
Great progress. I did find a bug I introduced when I modified the ipblacklist V2 perl script to add a space after the log-prefix BLKLST entry to make the logs compatible with other log-prefixes. This affected showrequestfromblacklist.dat and the modified version 'modified regex for V3 log-prefix added space' should be used.
https://git.ipfire.org/?p=people/helix/ipfire-2.x.git;a=commit;h=2ccc47f1944...
Good catch, but wouldn’t it be helpful to add the space to all log prefixes so that it is always easily readable for humans, too?
-Michael
I think ipblocklist was the only logfile entry to have this problem.
Rob
I think there is a missing space in rules.pl before tha last "" I think line 755 should be:
run("$IPTABLES -A ${blocklist}_DROP -j LOG -m limit --limit 10/second --log-prefix "BLKLST_$blocklist "");
Yes, that is the space character I am talking about.
and in my ipblocklists.dat I changed your line 298 to be compatible with core 163 as I don't think you can change the 'theme' now and produced an error.
from: &General::readhash("/srv/web/ipfire/html/themes/".$mainsettings{'THEME'}."/include/colors.txt", %color);
to: &General::readhash("/srv/web/ipfire/html/themes/ipfire/include/colors.txt", %color);
No, this has indeed been dropped and needs to be updated in the CGI. Let’s leave that to Stefan :)
-Michael
Rob
Hello Rob and list,
sorry for the long delay. Finally all major problems around the latest IDS improvements and the release around them has passed, so let's get back to the ipblocklist feature.
On Monday 07 March 2022 22:54 Rob Brewer wrote:
On Monday 07 March 2022 20:39 Michael Tremer wrote:
Hello Rob,
On 5 Mar 2022, at 21:46, Rob Brewer < ipfire-devel@grantura.co.uk> wrote:
Hi Stefan
On Saturday 05 March 2022 18:52 Stefan Schantl wrote:
Hello *,
I've made some development progress, which I want to share here:
Most parts of the main backend script ("ipblacklist") from Tim and Rob are ported into a new functions library (ipblocklist- functions.pl) and into the main firewall script (rules.pl).
Good
This process is almost finished and currently allows to create the firewall rules, download the blocklists and to convert them into an ipset compatible format.
Next step will be to import the frontend code (WUI) and adjust it to use the backend code (functions) from the "ipblocklist- functions.pl".
At this time the blocklist feature should be in a use-able state again and I'll go to create an automatic update script and to import all the logging pages stuff etc.
The development progress and single commits can be found here:
https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=shortlog;h=refs/hea...
As usual please feel free to ask any questions or to share your opinion here.
I wish you a nice day,
-Stefan
Great progress. I did find a bug I introduced when I modified the ipblacklist V2 perl script to add a space after the log-prefix BLKLST entry to make the logs compatible with other log-prefixes. This affected showrequestfromblacklist.dat and the modified version 'modified regex for V3 log-prefix added space' should be used.
https://git.ipfire.org/?p=people/helix/ipfire-2.x.git;a=commit;h=2ccc47f1944...
Thanks for the fix, I've picked it up and added it to the current development branch.
https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=80a456334f...
Good catch, but wouldn’t it be helpful to add the space to all log prefixes so that it is always easily readable for humans, too?
-Michael
I think ipblocklist was the only logfile entry to have this problem.
Rob
I think there is a missing space in rules.pl before tha last "" I think line 755 should be:
run("$IPTABLES -A ${blocklist}_DROP -j LOG -m limit --limit 10/second --log-prefix "BLKLST_$blocklist "");
Thanks for pointing this out - see:
https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=84642aadeb...
and in my ipblocklists.dat I changed your line 298 to be compatible with core 163 as I don't think you can change the 'theme' now and produced an error.
from: &General::readhash("/srv/web/ipfire/html/themes/".$mainsettings{'THE ME'}."/include/colors.txt", %color);
to: &General::readhash("/srv/web/ipfire/html/themes/ipfire/include/color s.txt", %color);
Fixed:
https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=acf4a99d23...
Rob
Hi Stefan,
Thank you for your update.
As you probably realize I have ben using Tim's IPblacklist software modified with the below changes for the past few weeks without problems so when it is released into IPFire, I wouldn't expect there to be any problems. Do you have an idea of the release issue you are aiming for for the IPFire version?
I have now integrated my ported Banish addon into IPblacklist which allows the user to maintain a personnel GUI based Banish list through the IPBlacklist interface which I find extremely useful. It updates considerably faster now it is using ipset rather than the original IPTables. I'll add the details to my git pages in the next few days.
Rob
On Sunday 03 April 2022 10:16 Stefan Schantl wrote:
Hello Rob and list,
sorry for the long delay. Finally all major problems around the latest IDS improvements and the release around them has passed, so let's get back to the ipblocklist feature.
On Monday 07 March 2022 22:54 Rob Brewer wrote:
On Monday 07 March 2022 20:39 Michael Tremer wrote:
Hello Rob,
On 5 Mar 2022, at 21:46, Rob Brewer < ipfire-devel@grantura.co.uk> wrote:
Hi Stefan
On Saturday 05 March 2022 18:52 Stefan Schantl wrote:
Hello *,
I've made some development progress, which I want to share here:
Most parts of the main backend script ("ipblacklist") from Tim and Rob are ported into a new functions library (ipblocklist- functions.pl) and into the main firewall script (rules.pl).
Good
This process is almost finished and currently allows to create the firewall rules, download the blocklists and to convert them into an ipset compatible format.
Next step will be to import the frontend code (WUI) and adjust it to use the backend code (functions) from the "ipblocklist- functions.pl".
At this time the blocklist feature should be in a use-able state again and I'll go to create an automatic update script and to import all the logging pages stuff etc.
The development progress and single commits can be found here:
https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=shortlog;h=refs/hea...
As usual please feel free to ask any questions or to share your opinion here.
I wish you a nice day,
-Stefan
Great progress. I did find a bug I introduced when I modified the ipblacklist V2 perl script to add a space after the log-prefix BLKLST entry to make the logs compatible with other log-prefixes. This affected showrequestfromblacklist.dat and the modified version 'modified regex for V3 log-prefix added space' should be used.
https://git.ipfire.org/?p=people/helix/ipfire-2.x.git;a=commit;h=2ccc47f1944...
Thanks for the fix, I've picked it up and added it to the current development branch.
https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=80a456334f...
Good catch, but wouldn’t it be helpful to add the space to all log prefixes so that it is always easily readable for humans, too?
-Michael
I think ipblocklist was the only logfile entry to have this problem.
Rob
I think there is a missing space in rules.pl before tha last "" I think line 755 should be:
run("$IPTABLES -A ${blocklist}_DROP -j LOG -m limit --limit 10/second --log-prefix "BLKLST_$blocklist "");
Thanks for pointing this out - see:
https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=84642aadeb...
and in my ipblocklists.dat I changed your line 298 to be compatible with core 163 as I don't think you can change the 'theme' now and produced an error.
from: &General::readhash("/srv/web/ipfire/html/themes/".$mainsettings{'THE ME'}."/include/colors.txt", %color);
to: &General::readhash("/srv/web/ipfire/html/themes/ipfire/include/color s.txt", %color);
Fixed:
https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=acf4a99d23...
Rob
-
Adolf Belka -
Michael Tremer -
Rob Brewer -
Stefan Schantl -
Tim FitzGeorge