IDS with support for multiple ruleset providers

IDS with support for multiple ruleset providers

[Stefan Schantl]

[-- Attachment #1: Type: text/plain, Size: 2150 bytes --]

Hello Development Team and list followers,

there are a lot of different vendors out there which offers different
IDS rules for suricata. Some of them offers a complete set of rules and
other ones some very specialized rules for different tasks.

Unfortunately it only was possible to select only one ruleset provider
at the same time, so it usually wasn't an option to use one of them and
keep a lot of traffic uninspected by the IDS.

Today I'm very happy to announce a testing version of a reworked
Intrusion Detection System which supports the usage of multiple
different providers and rulesets at the same time.

In total up to 15 different ruleset providers now can be used and mixed
together to fit your personal requirements. They easily can be managed
and configured via the WUI. Of course each one individually can be
disabled or re-enabled at each time.

The section for customizing the entire ruleset has been moved to a
subpage, which allows to enable a certain amount of ruleset files or
enabling / disabling single rules inside them.

This helps to speed up the CGI if you want to mange your whitelist,
manage your ruleset providers or change basic settings of your IDS. 

If you liked this short introduction, please help us testing to get
this cool stuff as soon as possible into the core distribution and to
find bugs or other improvements.

The test versions and some screenshots can be found here:

https://people.ipfire.org/~stevee/ids-multiple-providers/

To join testing, please download the latest tarball and place it on
your IPFire test machine.

Execute the archive by using "tar -xvf ids-multiple-providers-
XXX.tar.gz - C /" on your local console or via SSH remote session.

The next steps would be to regenerate the language cache by executing
"update-langs-cache" and to launch "convert-ids-multiple-providers".

The converter will convert all your existing settings into the new
format and also will take care about your used rules and their
settings.

As usual, please report back any kind of feedback on this list and
submit any found bugs to our bugtracker (https://bugs.ipfire.org).

Thanks in advance,

-Stefan

Re: IDS with support for multiple ruleset providers

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 4710 bytes --]

Hello,

I just tried to install this on my c156 test system and I have found the following issues:

The converter script was not executable:

[root(a)fw01 ~]# convert-ids-multiple-providers
-bash: /usr/sbin/convert-ids-multiple-providers: Permission denied

But that could be easily fixed:

[root(a)fw01 ~]# chmod a+x /usr/sbin/convert-ids-multiple-providers

But then it fails with another error; probably a typo:

[root(a)fw01 ~]# convert-ids-multiple-providers
Can't locate /var/ipfire/ids-functions.pl1 at /usr/sbin/convert-ids-multiple-providers line 25.

Fixing that gives me this:

[root(a)fw01 ~]# convert-ids-multiple-providers
Could not write to /var/ipfire/suricata/oinkmaster-emerging-modified-sids.conf. No such file or directory

Creating that file makes the converter exit without any errors:

[root(a)fw01 ~]# touch /var/ipfire/suricata/oinkmaster-emerging-modified-sids.conf
[root(a)fw01 ~]# convert-ids-multiple-providers
[root(a)fw01 ~]#

But it didn’t convert anything and the file is empty.

However, the CGI loads and shows the new functionality.

Clicking the checkbox to disable a ruleset shows this:

Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
Could not write to /var/ipfire/suricata/suricata-used-providers.yaml. Permission denied

That file exists but is owned by root:

-rw-r--r-- 1 root   root       0 Apr 10 13:42 suricata-used-providers.yaml

Changing permissions to nobody:nobody brings me one step further:

Could not write to /var/ipfire/suricata/oinkmaster-provider-includes.conf. Permission denied

After fixing the ownership, I can enable and disable the rule provider.

However, when I click “customise ruleset”, no rules are selected which suggests that the converter did not work correctly.

I could also add a new provider and found it confusing that no categories are enabled automatically. It could happen, that you add a couple of providers, but then there are still no rules enabled. That would be potentially dangerous. Thoughts on this?

Apart from that, this seems to work solidly even with the new perl release.

-Michael

> On 9 Apr 2021, at 20:27, Stefan Schantl <stefan.schantl(a)ipfire.org> wrote:
> 
> Hello Development Team and list followers,
> 
> there are a lot of different vendors out there which offers different
> IDS rules for suricata. Some of them offers a complete set of rules and
> other ones some very specialized rules for different tasks.
> 
> Unfortunately it only was possible to select only one ruleset provider
> at the same time, so it usually wasn't an option to use one of them and
> keep a lot of traffic uninspected by the IDS.
> 
> Today I'm very happy to announce a testing version of a reworked
> Intrusion Detection System which supports the usage of multiple
> different providers and rulesets at the same time.
> 
> In total up to 15 different ruleset providers now can be used and mixed
> together to fit your personal requirements. They easily can be managed
> and configured via the WUI. Of course each one individually can be
> disabled or re-enabled at each time.
> 
> The section for customizing the entire ruleset has been moved to a
> subpage, which allows to enable a certain amount of ruleset files or
> enabling / disabling single rules inside them.
> 
> This helps to speed up the CGI if you want to mange your whitelist,
> manage your ruleset providers or change basic settings of your IDS. 
> 
> If you liked this short introduction, please help us testing to get
> this cool stuff as soon as possible into the core distribution and to
> find bugs or other improvements.
> 
> The test versions and some screenshots can be found here:
> 
> https://people.ipfire.org/~stevee/ids-multiple-providers/
> 
> To join testing, please download the latest tarball and place it on
> your IPFire test machine.
> 
> Execute the archive by using "tar -xvf ids-multiple-providers-
> XXX.tar.gz - C /" on your local console or via SSH remote session.
> 
> The next steps would be to regenerate the language cache by executing
> "update-langs-cache" and to launch "convert-ids-multiple-providers".
> 
> The converter will convert all your existing settings into the new
> format and also will take care about your used rules and their
> settings.
> 
> As usual, please report back any kind of feedback on this list and
> submit any found bugs to our bugtracker (https://bugs.ipfire.org).
> 
> Thanks in advance,
> 
> -Stefan
> 
> 

Re: IDS with support for multiple ruleset providers

[Stefan Schantl]

[-- Attachment #1: Type: text/plain, Size: 5848 bytes --]

Hello Michael,

a big thanks for testing and providing a lot of feedback and finding
all this bugs.

> Hello,
> 
> I just tried to install this on my c156 test system and I have found
> the following issues:
> 
> The converter script was not executable:
> 
> [root(a)fw01 ~]# convert-ids-multiple-providers
> -bash: /usr/sbin/convert-ids-multiple-providers: Permission denied
> 
> But that could be easily fixed:

Thanks, I've fixed this in the 003 release.
> 
> [root(a)fw01 ~]# chmod a+x /usr/sbin/convert-ids-multiple-providers
> 
> But then it fails with another error; probably a typo:
> 
> [root(a)fw01 ~]# convert-ids-multiple-providers
> Can't locate /var/ipfire/ids-functions.pl1 at /usr/sbin/convert-ids-
> multiple-providers line 25.

Shame on me, this was during testing while developing to have the old
and the changed ids-functions.pl on the same system. Accidently I've
commited the file with this hack and shipped it.

Because both libs were present on my development system I didn't note
about this problem.

> 
> Fixing that gives me this:
> 
> [root(a)fw01 ~]# convert-ids-multiple-providers
> Could not write to /var/ipfire/suricata/oinkmaster-emerging-modified-
> sids.conf. No such file or directory
> 
> Creating that file makes the converter exit without any errors:
> 

Yea, this happened, because I didn't proper opened the file for writing
in the converter script. So It has not been created and the old ruleset
changes also are not written to it.

> [root(a)fw01 ~]# touch /var/ipfire/suricata/oinkmaster-emerging-
> modified-sids.conf
> [root(a)fw01 ~]# convert-ids-multiple-providers
> [root(a)fw01 ~]#
> 
> But it didn’t convert anything and the file is empty.
> 
> However, the CGI loads and shows the new functionality.
> 
> Clicking the checkbox to disable a ruleset shows this:
> 
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> 288.
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> 288.
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> 288.
> Could not write to /var/ipfire/suricata/suricata-used-providers.yaml. 
> Permission denied
> 
> That file exists but is owned by root:
> 
> -rw-r--r-- 1 root   root       0 Apr 10 13:42 suricata-used-
> providers.yaml

Strange, the ownerships of these files should be changed to the correct
one, by the converter script. I'll have a look on it. 

> 
> Changing permissions to nobody:nobody brings me one step further:
> 
> Could not write to /var/ipfire/suricata/oinkmaster-provider-
> includes.conf. Permission denied
> 
> After fixing the ownership, I can enable and disable the rule
> provider.
> 
> However, when I click “customise ruleset”, no rules are selected
> which suggests that the converter did not work correctly.

See above.

> 
> I could also add a new provider and found it confusing that no
> categories are enabled automatically. It could happen, that you add a
> couple of providers, but then there are still no rules enabled. That
> would be potentially dangerous. Thoughts on this?

The easiest solution would be to check if at least one ruleset provider
is enabled and has at least one rulefile enabled.

This would prevent from starting suricata without any rules.

> 
> Apart from that, this seems to work solidly even with the new perl
> release.
> 
> -Michael
> 
> > On 9 Apr 2021, at 20:27, Stefan Schantl <stefan.schantl(a)ipfire.org>
> > wrote:
> > 
> > Hello Development Team and list followers,
> > 
> > there are a lot of different vendors out there which offers
> > different
> > IDS rules for suricata. Some of them offers a complete set of rules
> > and
> > other ones some very specialized rules for different tasks.
> > 
> > Unfortunately it only was possible to select only one ruleset
> > provider
> > at the same time, so it usually wasn't an option to use one of them
> > and
> > keep a lot of traffic uninspected by the IDS.
> > 
> > Today I'm very happy to announce a testing version of a reworked
> > Intrusion Detection System which supports the usage of multiple
> > different providers and rulesets at the same time.
> > 
> > In total up to 15 different ruleset providers now can be used and
> > mixed
> > together to fit your personal requirements. They easily can be
> > managed
> > and configured via the WUI. Of course each one individually can be
> > disabled or re-enabled at each time.
> > 
> > The section for customizing the entire ruleset has been moved to a
> > subpage, which allows to enable a certain amount of ruleset files
> > or
> > enabling / disabling single rules inside them.
> > 
> > This helps to speed up the CGI if you want to mange your whitelist,
> > manage your ruleset providers or change basic settings of your
> > IDS. 
> > 
> > If you liked this short introduction, please help us testing to get
> > this cool stuff as soon as possible into the core distribution and
> > to
> > find bugs or other improvements.
> > 
> > The test versions and some screenshots can be found here:
> > 
> > https://people.ipfire.org/~stevee/ids-multiple-providers/
> > 
> > To join testing, please download the latest tarball and place it on
> > your IPFire test machine.
> > 
> > Execute the archive by using "tar -xvf ids-multiple-providers-
> > XXX.tar.gz - C /" on your local console or via SSH remote session.
> > 
> > The next steps would be to regenerate the language cache by
> > executing
> > "update-langs-cache" and to launch "convert-ids-multiple-
> > providers".
> > 
> > The converter will convert all your existing settings into the new
> > format and also will take care about your used rules and their
> > settings.
> > 
> > As usual, please report back any kind of feedback on this list and
> > submit any found bugs to our bugtracker (https://bugs.ipfire.org).
> > 
> > Thanks in advance,
> > 
> > -Stefan
> > 
> > 

Re: IDS with support for multiple ruleset providers

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 3836 bytes --]

Hi Stefan,

I tested this on my vm testbed.

On 09/04/2021 21:27, Stefan Schantl wrote:
> Hello Development Team and list followers,
>
> there are a lot of different vendors out there which offers different
> IDS rules for suricata. Some of them offers a complete set of rules and
> other ones some very specialized rules for different tasks.
>
> Unfortunately it only was possible to select only one ruleset provider
> at the same time, so it usually wasn't an option to use one of them and
> keep a lot of traffic uninspected by the IDS.
>
> Today I'm very happy to announce a testing version of a reworked
> Intrusion Detection System which supports the usage of multiple
> different providers and rulesets at the same time.
>
> In total up to 15 different ruleset providers now can be used and mixed
> together to fit your personal requirements. They easily can be managed
> and configured via the WUI. Of course each one individually can be
> disabled or re-enabled at each time.
>
> The section for customizing the entire ruleset has been moved to a
> subpage, which allows to enable a certain amount of ruleset files or
> enabling / disabling single rules inside them.
>
> This helps to speed up the CGI if you want to mange your whitelist,
> manage your ruleset providers or change basic settings of your IDS.
>
> If you liked this short introduction, please help us testing to get
> this cool stuff as soon as possible into the core distribution and to
> find bugs or other improvements.
>
> The test versions and some screenshots can be found here:
>
> https://people.ipfire.org/~stevee/ids-multiple-providers/
>
> To join testing, please download the latest tarball and place it on
> your IPFire test machine.
>
> Execute the archive by using "tar -xvf ids-multiple-providers-
> XXX.tar.gz - C /" on your local console or via SSH remote session.
> bash: /usr/sbin/convert-ids-multiple-providers: Permission denied
Extracting the archive worked with no problems.
> The next steps would be to regenerate the language cache by executing
> "update-langs-cache" and to launch "convert-ids-multiple-providers".

update-lang-cache worked fine. When tried to run convert-ids-multiple-providers I got the message

bash: /usr/sbin/convert-ids-multiple-providers: Permission denied

I was running the command as root so I checked the file and it was not set as executable. I changed this and it then ran but came back with the following error message

Can't locate /var/ipfire/ids-functions.pl1 at /usr/sbin/convert-ids-multiple-providers line 25

I edited the .pl1 to .pl and re-ran the converter and it completed without any further error message.


I then had the new WUI IDS page.


I selected an additional provider, OISF, and it was added to the list of providers. I then selected customise rules and I selected the oisf ruleset and pressed apply. I just got a white screen with nothing happening. I then reloaded IPFire in the browser again and OISF provider was still listed but on the rules page it was not selected. Tried again and same thing happened. I then pressed the delete button to remove the OISF provider from the list and I get the message "The ruleset changes are being applied. Please wait until all opersations have completed successfully..." That message has not changed since I started writing this email. I then reloaded IPFire in the browser and OISF had been removed from the list.


Regards,

Adolf

> The converter will convert all your existing settings into the new
> format and also will take care about your used rules and their
> settings.
>
> As usual, please report back any kind of feedback on this list and
> submit any found bugs to our bugtracker (https://bugs.ipfire.org).
>
> Thanks in advance,
>
> -Stefan
>
>

Re: IDS with support for multiple ruleset providers

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 4296 bytes --]

Hi Stefan,

On 10/04/2021 15:06, Adolf Belka wrote:
> Hi Stefan,
>
> I tested this on my vm testbed.
>
> On 09/04/2021 21:27, Stefan Schantl wrote:
>> Hello Development Team and list followers,
>>
>> there are a lot of different vendors out there which offers different
>> IDS rules for suricata. Some of them offers a complete set of rules and
>> other ones some very specialized rules for different tasks.
>>
>> Unfortunately it only was possible to select only one ruleset provider
>> at the same time, so it usually wasn't an option to use one of them and
>> keep a lot of traffic uninspected by the IDS.
>>
>> Today I'm very happy to announce a testing version of a reworked
>> Intrusion Detection System which supports the usage of multiple
>> different providers and rulesets at the same time.
>>
>> In total up to 15 different ruleset providers now can be used and mixed
>> together to fit your personal requirements. They easily can be managed
>> and configured via the WUI. Of course each one individually can be
>> disabled or re-enabled at each time.
>>
>> The section for customizing the entire ruleset has been moved to a
>> subpage, which allows to enable a certain amount of ruleset files or
>> enabling / disabling single rules inside them.
>>
>> This helps to speed up the CGI if you want to mange your whitelist,
>> manage your ruleset providers or change basic settings of your IDS.
>>
>> If you liked this short introduction, please help us testing to get
>> this cool stuff as soon as possible into the core distribution and to
>> find bugs or other improvements.
>>
>> The test versions and some screenshots can be found here:
>>
>> https://people.ipfire.org/~stevee/ids-multiple-providers/
>>
>> To join testing, please download the latest tarball and place it on
>> your IPFire test machine.
>>
>> Execute the archive by using "tar -xvf ids-multiple-providers-
>> XXX.tar.gz - C /" on your local console or via SSH remote session.
>> bash: /usr/sbin/convert-ids-multiple-providers: Permission denied
> Extracting the archive worked with no problems.
>> The next steps would be to regenerate the language cache by executing
>> "update-langs-cache" and to launch "convert-ids-multiple-providers".
>
> update-lang-cache worked fine. When tried to run convert-ids-multiple-providers I got the message
>
> bash: /usr/sbin/convert-ids-multiple-providers: Permission denied
>
> I was running the command as root so I checked the file and it was not set as executable. I changed this and it then ran but came back with the following error message
>
> Can't locate /var/ipfire/ids-functions.pl1 at /usr/sbin/convert-ids-multiple-providers line 25
>
> I edited the .pl1 to .pl and re-ran the converter and it completed without any further error message.
>
>
> I then had the new WUI IDS page.
>
>
> I selected an additional provider, OISF, and it was added to the list of providers. I then selected customise rules and I selected the oisf ruleset and pressed apply. I just got a white screen with nothing happening. I then reloaded IPFire in the browser again and OISF provider was still listed but on the rules page it was not selected. Tried again and same thing happened. I then pressed the delete button to remove the OISF provider from the list and I get the message "The ruleset changes are being applied. Please wait until all opersations have completed successfully..." That message has not changed since I started writing this email. I then reloaded IPFire in the browser and OISF had been removed from the list.
>
>
Here are the error messages from my httpd/error_log file

Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
Could not write to /var/ipfire/suricata/oinkmaster-provider-includes.conf. Permission denied
Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288

Adolf

> Regards,
>
> Adolf
>
>> The converter will convert all your existing settings into the new
>> format and also will take care about your used rules and their
>> settings.
>>
>> As usual, please report back any kind of feedback on this list and
>> submit any found bugs to our bugtracker (https://bugs.ipfire.org).
>>
>> Thanks in advance,
>>
>> -Stefan
>>
>>

Re: IDS with support for multiple ruleset providers

[Stefan Schantl]

[-- Attachment #1: Type: text/plain, Size: 4312 bytes --]

Hello Adolf,

a big thanks for downloading, testing and sharing your feedback.

As you reported mostly the same issues than Michael did, I've answered
them already in the response to his mail.

I'm going to fix all this issues and report back if a new test version
is finished.

Best regards,

- Stefan

> Hi Stefan,
> 
> I tested this on my vm testbed.
> 
> On 09/04/2021 21:27, Stefan Schantl wrote:
> > Hello Development Team and list followers,
> > 
> > there are a lot of different vendors out there which offers
> > different
> > IDS rules for suricata. Some of them offers a complete set of rules
> > and
> > other ones some very specialized rules for different tasks.
> > 
> > Unfortunately it only was possible to select only one ruleset
> > provider
> > at the same time, so it usually wasn't an option to use one of them
> > and
> > keep a lot of traffic uninspected by the IDS.
> > 
> > Today I'm very happy to announce a testing version of a reworked
> > Intrusion Detection System which supports the usage of multiple
> > different providers and rulesets at the same time.
> > 
> > In total up to 15 different ruleset providers now can be used and
> > mixed
> > together to fit your personal requirements. They easily can be
> > managed
> > and configured via the WUI. Of course each one individually can be
> > disabled or re-enabled at each time.
> > 
> > The section for customizing the entire ruleset has been moved to a
> > subpage, which allows to enable a certain amount of ruleset files
> > or
> > enabling / disabling single rules inside them.
> > 
> > This helps to speed up the CGI if you want to mange your whitelist,
> > manage your ruleset providers or change basic settings of your IDS.
> > 
> > If you liked this short introduction, please help us testing to get
> > this cool stuff as soon as possible into the core distribution and
> > to
> > find bugs or other improvements.
> > 
> > The test versions and some screenshots can be found here:
> > 
> > https://people.ipfire.org/~stevee/ids-multiple-providers/
> > 
> > To join testing, please download the latest tarball and place it on
> > your IPFire test machine.
> > 
> > Execute the archive by using "tar -xvf ids-multiple-providers-
> > XXX.tar.gz - C /" on your local console or via SSH remote session.
> > bash: /usr/sbin/convert-ids-multiple-providers: Permission denied
> Extracting the archive worked with no problems.
> > The next steps would be to regenerate the language cache by
> > executing
> > "update-langs-cache" and to launch "convert-ids-multiple-
> > providers".
> 
> update-lang-cache worked fine. When tried to run convert-ids-
> multiple-providers I got the message
> 
> bash: /usr/sbin/convert-ids-multiple-providers: Permission denied
> 
> I was running the command as root so I checked the file and it was
> not set as executable. I changed this and it then ran but came back
> with the following error message
> 
> Can't locate /var/ipfire/ids-functions.pl1 at /usr/sbin/convert-ids-
> multiple-providers line 25
> 
> I edited the .pl1 to .pl and re-ran the converter and it completed
> without any further error message.
> 
> 
> I then had the new WUI IDS page.
> 
> 
> I selected an additional provider, OISF, and it was added to the list
> of providers. I then selected customise rules and I selected the oisf
> ruleset and pressed apply. I just got a white screen with nothing
> happening. I then reloaded IPFire in the browser again and OISF
> provider was still listed but on the rules page it was not selected.
> Tried again and same thing happened. I then pressed the delete button
> to remove the OISF provider from the list and I get the message "The
> ruleset changes are being applied. Please wait until all opersations
> have completed successfully..." That message has not changed since I
> started writing this email. I then reloaded IPFire in the browser and
> OISF had been removed from the list.
> 
> 
> Regards,
> 
> Adolf
> 
> > The converter will convert all your existing settings into the new
> > format and also will take care about your used rules and their
> > settings.
> > 
> > As usual, please report back any kind of feedback on this list and
> > submit any found bugs to our bugtracker (https://bugs.ipfire.org).
> > 
> > Thanks in advance,
> > 
> > -Stefan
> > 
> > 

Re: IDS with support for multiple ruleset providers

[Stefan Schantl]

[-- Attachment #1: Type: text/plain, Size: 591 bytes --]

Hello list followers,

after getting a lot of feedback and bug reports I'm happy to
announce the third test version for the new IDS system.

https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-003.tar.gz

If you just join testing, please omit the installation instructions
from the initial Mail from this list.

The converter script now works as expected and runs very smooth.

As usual please post your feedback and opinions to this list and any
remain bugs to our bugtracker. (https://bugzilla.ipfire.org)

A big thanks in advance,

-Stefan

Re: IDS with support for multiple ruleset providers

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 3128 bytes --]

Hi Stefan,

I copied the new tarfile to my ipfire vm testbed machine and extracted it and ran the converter script. No errors. I then used the wui page to add a new provider to the list then selected to customize the rules and ticked the box for the added rules. Then I pressed apply and got a blank white screen again.


The error log has the following:-

Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
Could not open /var/ipfire/suricata/oinkmaster-provider-includes.conf. Permission denied


ls- hal of /var/ipfire/suricata shows the following

drwxr-xr-x  2 nobody nobody 4.0K Apr 10 22:47 .
drwxr-xr-x 49 root   root   4.0K Apr  5 08:20 ..
-rw-r--r--  1 nobody nobody    0 Dec 14 19:05 ignored
-rw-r--r--  1 root   root    21K Apr  1 20:00 oinkmaster.conf
-rw-r--r--  1 nobody nobody   61 Apr 10 14:40 oinkmaster-modify-sids.conf
-rw-r--r--  1 root   root      0 Apr 10 14:54 oinkmaster-provider-includes.conf
-rw-r--r--  1 nobody nobody   55 Apr 10 22:47 providers-settings
-rw-r--r--  1 root   root   6.0K Apr  5 07:13 ruleset-sources
-rw-r--r--  1 nobody nobody  102 Apr 10 14:54 settings
-rw-r--r--  1 nobody nobody  140 Apr 10 22:41 suricata-dns-servers.yaml
-rw-r--r--  1 nobody nobody  125 Apr 10 14:54 suricata-emerging-used-rulefiles.yaml
-rw-r--r--  1 nobody nobody  159 Apr 10 22:41 suricata-homenet.yaml
-rw-r--r--  1 nobody nobody   98 Apr 10 14:40 suricata-http-ports.yaml
-rw-r--r--  1 nobody nobody   95 Apr 10 14:54 suricata-static-included-rulefiles.yaml
-rw-r--r--  1 nobody nobody   76 Apr 10 22:47 suricata-urlhaus-used-rulefiles.yaml
-rw-r--r--  1 nobody nobody  214 Apr 10 14:54 suricata-used-providers.yaml

Three of the files are owned root:root while all the others are nobody:nobody


The above was with extracting and applying the updated tar file on top of IPFire after running the last version.

I will do a fresh clone of my IPFire vm and then repeat the tar extraction and convert and see if that gives any difference.


Regards,

Adolf

On 10/04/2021 20:25, Stefan Schantl wrote:
> Hello list followers,
>
> after getting a lot of feedback and bug reports I'm happy to
> announce the third test version for the new IDS system.
>
> https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-003.tar.gz
>
> If you just join testing, please omit the installation instructions
> from the initial Mail from this list.
>
> The converter script now works as expected and runs very smooth.
>
> As usual please post your feedback and opinions to this list and any
> remain bugs to our bugtracker. (https://bugzilla.ipfire.org)
>
> A big thanks in advance,
>
> -Stefan
>

Re: IDS with support for multiple ruleset providers

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 4246 bytes --]

Hi Stefan,

I did a fresh install of the latest tar file and ran the convert script. It ran for a bit longer than in the past and then stopped with no errors.

I then went to the WUI page and it showed "Downloading and unpacking new ruleset. Please wait until all operations have completed successfully..."

It is still showing that message after more than 5 minutes and the error log has a large number of the following lines in it:-

Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.

The number of lines keeps increasing with time so it seems something is in a loop. So this time I never even got to see the IDS WUI page. Reloading the IPFire browser and re-selecting IDS gives the same message.


Regards,

Adolf.

On 10/04/2021 22:56, Adolf Belka wrote:
> Hi Stefan,
>
> I copied the new tarfile to my ipfire vm testbed machine and extracted it and ran the converter script. No errors. I then used the wui page to add a new provider to the list then selected to customize the rules and ticked the box for the added rules. Then I pressed apply and got a blank white screen again.
>
>
> The error log has the following:-
>
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
> Could not open /var/ipfire/suricata/oinkmaster-provider-includes.conf. Permission denied
>
>
> ls- hal of /var/ipfire/suricata shows the following
>
> drwxr-xr-x  2 nobody nobody 4.0K Apr 10 22:47 .
> drwxr-xr-x 49 root   root   4.0K Apr  5 08:20 ..
> -rw-r--r--  1 nobody nobody    0 Dec 14 19:05 ignored
> -rw-r--r--  1 root   root    21K Apr  1 20:00 oinkmaster.conf
> -rw-r--r--  1 nobody nobody   61 Apr 10 14:40 oinkmaster-modify-sids.conf
> -rw-r--r--  1 root   root      0 Apr 10 14:54 oinkmaster-provider-includes.conf
> -rw-r--r--  1 nobody nobody   55 Apr 10 22:47 providers-settings
> -rw-r--r--  1 root   root   6.0K Apr  5 07:13 ruleset-sources
> -rw-r--r--  1 nobody nobody  102 Apr 10 14:54 settings
> -rw-r--r--  1 nobody nobody  140 Apr 10 22:41 suricata-dns-servers.yaml
> -rw-r--r--  1 nobody nobody  125 Apr 10 14:54 suricata-emerging-used-rulefiles.yaml
> -rw-r--r--  1 nobody nobody  159 Apr 10 22:41 suricata-homenet.yaml
> -rw-r--r--  1 nobody nobody   98 Apr 10 14:40 suricata-http-ports.yaml
> -rw-r--r--  1 nobody nobody   95 Apr 10 14:54 suricata-static-included-rulefiles.yaml
> -rw-r--r--  1 nobody nobody   76 Apr 10 22:47 suricata-urlhaus-used-rulefiles.yaml
> -rw-r--r--  1 nobody nobody  214 Apr 10 14:54 suricata-used-providers.yaml
>
> Three of the files are owned root:root while all the others are nobody:nobody
>
>
> The above was with extracting and applying the updated tar file on top of IPFire after running the last version.
>
> I will do a fresh clone of my IPFire vm and then repeat the tar extraction and convert and see if that gives any difference.
>
>
> Regards,
>
> Adolf
>
> On 10/04/2021 20:25, Stefan Schantl wrote:
>> Hello list followers,
>>
>> after getting a lot of feedback and bug reports I'm happy to
>> announce the third test version for the new IDS system.
>>
>> https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-003.tar.gz
>>
>> If you just join testing, please omit the installation instructions
>> from the initial Mail from this list.
>>
>> The converter script now works as expected and runs very smooth.
>>
>> As usual please post your feedback and opinions to this list and any
>> remain bugs to our bugtracker. (https://bugzilla.ipfire.org)
>>
>> A big thanks in advance,
>>
>> -Stefan
>>

Re: IDS with support for multiple ruleset providers

[Stefan Schantl]

[-- Attachment #1: Type: text/plain, Size: 4774 bytes --]

Good morning Adolf,

you missed to update the language cache after extracting the archive,
so the language strings are missing and the WUI infinite loops here.

Best regards,

-Stefan
 
> Hi Stefan,
> 
> I did a fresh install of the latest tar file and ran the convert
> script. It ran for a bit longer than in the past and then stopped
> with no errors.
> 
> I then went to the WUI page and it showed "Downloading and unpacking
> new ruleset. Please wait until all operations have completed
> successfully..."
> 
> It is still showing that message after more than 5 minutes and the
> error log has a large number of the following lines in it:-
> 
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> 288.
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> 288.
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> 288.
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> 288.
> 
> The number of lines keeps increasing with time so it seems something
> is in a loop. So this time I never even got to see the IDS WUI page.
> Reloading the IPFire browser and re-selecting IDS gives the same
> message.
> 
> 
> Regards,
> 
> Adolf.
> 
> On 10/04/2021 22:56, Adolf Belka wrote:
> > Hi Stefan,
> > 
> > I copied the new tarfile to my ipfire vm testbed machine and
> > extracted it and ran the converter script. No errors. I then used
> > the wui page to add a new provider to the list then selected to
> > customize the rules and ticked the box for the added rules. Then I
> > pressed apply and got a blank white screen again.
> > 
> > 
> > The error log has the following:-
> > 
> > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> > 288.
> > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> > 288.
> > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> > 288.
> > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> > 288.
> > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> > 288.
> > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> > 288.
> > Could not open /var/ipfire/suricata/oinkmaster-provider-
> > includes.conf. Permission denied
> > 
> > 
> > ls- hal of /var/ipfire/suricata shows the following
> > 
> > drwxr-xr-x  2 nobody nobody 4.0K Apr 10 22:47 .
> > drwxr-xr-x 49 root   root   4.0K Apr  5 08:20 ..
> > -rw-r--r--  1 nobody nobody    0 Dec 14 19:05 ignored
> > -rw-r--r--  1 root   root    21K Apr  1 20:00 oinkmaster.conf
> > -rw-r--r--  1 nobody nobody   61 Apr 10 14:40 oinkmaster-modify-
> > sids.conf
> > -rw-r--r--  1 root   root      0 Apr 10 14:54 oinkmaster-provider-
> > includes.conf
> > -rw-r--r--  1 nobody nobody   55 Apr 10 22:47 providers-settings
> > -rw-r--r--  1 root   root   6.0K Apr  5 07:13 ruleset-sources
> > -rw-r--r--  1 nobody nobody  102 Apr 10 14:54 settings
> > -rw-r--r--  1 nobody nobody  140 Apr 10 22:41 suricata-dns-
> > servers.yaml
> > -rw-r--r--  1 nobody nobody  125 Apr 10 14:54 suricata-emerging-
> > used-rulefiles.yaml
> > -rw-r--r--  1 nobody nobody  159 Apr 10 22:41 suricata-homenet.yaml
> > -rw-r--r--  1 nobody nobody   98 Apr 10 14:40 suricata-http-
> > ports.yaml
> > -rw-r--r--  1 nobody nobody   95 Apr 10 14:54 suricata-static-
> > included-rulefiles.yaml
> > -rw-r--r--  1 nobody nobody   76 Apr 10 22:47 suricata-urlhaus-
> > used-rulefiles.yaml
> > -rw-r--r--  1 nobody nobody  214 Apr 10 14:54 suricata-used-
> > providers.yaml
> > 
> > Three of the files are owned root:root while all the others are
> > nobody:nobody
> > 
> > 
> > The above was with extracting and applying the updated tar file on
> > top of IPFire after running the last version.
> > 
> > I will do a fresh clone of my IPFire vm and then repeat the tar
> > extraction and convert and see if that gives any difference.
> > 
> > 
> > Regards,
> > 
> > Adolf
> > 
> > On 10/04/2021 20:25, Stefan Schantl wrote:
> > > Hello list followers,
> > > 
> > > after getting a lot of feedback and bug reports I'm happy to
> > > announce the third test version for the new IDS system.
> > > 
> > > https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-003.tar.gz
> > > 
> > > If you just join testing, please omit the installation
> > > instructions
> > > from the initial Mail from this list.
> > > 
> > > The converter script now works as expected and runs very smooth.
> > > 
> > > As usual please post your feedback and opinions to this list and
> > > any
> > > remain bugs to our bugtracker. (https://bugzilla.ipfire.org)
> > > 
> > > A big thanks in advance,
> > > 
> > > -Stefan
> > > 

Re: IDS with support for multiple ruleset providers

[Stefan Schantl]

[-- Attachment #1: Type: text/plain, Size: 3758 bytes --]

Good morning Adolf,

thanks for re-testing and reporting this issue.

After digging through the code I finally found why the file permission
on your test system was different to mine.

In the if clause, the ownership of the file only would be changed if
ruleset changes had been made, which of course is wrong here.

I'll upload a new testversion very soon.

Once again, a big thanks for testing and pointing this out.

Best regards,

-Stefan

> Hi Stefan,
> 
> I copied the new tarfile to my ipfire vm testbed machine and
> extracted it and ran the converter script. No errors. I then used the
> wui page to add a new provider to the list then selected to customize
> the rules and ticked the box for the added rules. Then I pressed
> apply and got a blank white screen again.
> 
> 
> The error log has the following:-
> 
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> 288.
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> 288.
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> 288.
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> 288.
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> 288.
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> 288.
> Could not open /var/ipfire/suricata/oinkmaster-provider-
> includes.conf. Permission denied
> 
> 
> ls- hal of /var/ipfire/suricata shows the following
> 
> drwxr-xr-x  2 nobody nobody 4.0K Apr 10 22:47 .
> drwxr-xr-x 49 root   root   4.0K Apr  5 08:20 ..
> -rw-r--r--  1 nobody nobody    0 Dec 14 19:05 ignored
> -rw-r--r--  1 root   root    21K Apr  1 20:00 oinkmaster.conf
> -rw-r--r--  1 nobody nobody   61 Apr 10 14:40 oinkmaster-modify-
> sids.conf
> -rw-r--r--  1 root   root      0 Apr 10 14:54 oinkmaster-provider-
> includes.conf
> -rw-r--r--  1 nobody nobody   55 Apr 10 22:47 providers-settings
> -rw-r--r--  1 root   root   6.0K Apr  5 07:13 ruleset-sources
> -rw-r--r--  1 nobody nobody  102 Apr 10 14:54 settings
> -rw-r--r--  1 nobody nobody  140 Apr 10 22:41 suricata-dns-
> servers.yaml
> -rw-r--r--  1 nobody nobody  125 Apr 10 14:54 suricata-emerging-used-
> rulefiles.yaml
> -rw-r--r--  1 nobody nobody  159 Apr 10 22:41 suricata-homenet.yaml
> -rw-r--r--  1 nobody nobody   98 Apr 10 14:40 suricata-http-
> ports.yaml
> -rw-r--r--  1 nobody nobody   95 Apr 10 14:54 suricata-static-
> included-rulefiles.yaml
> -rw-r--r--  1 nobody nobody   76 Apr 10 22:47 suricata-urlhaus-used-
> rulefiles.yaml
> -rw-r--r--  1 nobody nobody  214 Apr 10 14:54 suricata-used-
> providers.yaml
> 
> Three of the files are owned root:root while all the others are
> nobody:nobody
> 
> 
> The above was with extracting and applying the updated tar file on
> top of IPFire after running the last version.
> 
> I will do a fresh clone of my IPFire vm and then repeat the tar
> extraction and convert and see if that gives any difference.
> 
> 
> Regards,
> 
> Adolf
> 
> On 10/04/2021 20:25, Stefan Schantl wrote:
> > Hello list followers,
> > 
> > after getting a lot of feedback and bug reports I'm happy to
> > announce the third test version for the new IDS system.
> > 
> > https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-003.tar.gz
> > 
> > If you just join testing, please omit the installation instructions
> > from the initial Mail from this list.
> > 
> > The converter script now works as expected and runs very smooth.
> > 
> > As usual please post your feedback and opinions to this list and
> > any
> > remain bugs to our bugtracker. (https://bugzilla.ipfire.org)
> > 
> > A big thanks in advance,
> > 
> > -Stefan
> > 

Re: IDS with support for multiple ruleset providers

[Stefan Schantl]

[-- Attachment #1: Type: text/plain, Size: 3849 bytes --]

Hello again,

I've tested and uploaded the fourth test verstion.

https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-004.tar.gz

This time the ownership of all files are correct at my test system.

(Tested with ruleset changes and without)

Best regards,

-Stefan

> Best regards,
> 
> -Stefan
> 
> > Hi Stefan,
> > 
> > I copied the new tarfile to my ipfire vm testbed machine and
> > extracted it and ran the converter script. No errors. I then used
> > the
> > wui page to add a new provider to the list then selected to
> > customize
> > the rules and ticked the box for the added rules. Then I pressed
> > apply and got a blank white screen again.
> > 
> > 
> > The error log has the following:-
> > 
> > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> > 288.
> > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> > 288.
> > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> > 288.
> > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> > 288.
> > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> > 288.
> > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> > 288.
> > Could not open /var/ipfire/suricata/oinkmaster-provider-
> > includes.conf. Permission denied
> > 
> > 
> > ls- hal of /var/ipfire/suricata shows the following
> > 
> > drwxr-xr-x  2 nobody nobody 4.0K Apr 10 22:47 .
> > drwxr-xr-x 49 root   root   4.0K Apr  5 08:20 ..
> > -rw-r--r--  1 nobody nobody    0 Dec 14 19:05 ignored
> > -rw-r--r--  1 root   root    21K Apr  1 20:00 oinkmaster.conf
> > -rw-r--r--  1 nobody nobody   61 Apr 10 14:40 oinkmaster-modify-
> > sids.conf
> > -rw-r--r--  1 root   root      0 Apr 10 14:54 oinkmaster-provider-
> > includes.conf
> > -rw-r--r--  1 nobody nobody   55 Apr 10 22:47 providers-settings
> > -rw-r--r--  1 root   root   6.0K Apr  5 07:13 ruleset-sources
> > -rw-r--r--  1 nobody nobody  102 Apr 10 14:54 settings
> > -rw-r--r--  1 nobody nobody  140 Apr 10 22:41 suricata-dns-
> > servers.yaml
> > -rw-r--r--  1 nobody nobody  125 Apr 10 14:54 suricata-emerging-
> > used-
> > rulefiles.yaml
> > -rw-r--r--  1 nobody nobody  159 Apr 10 22:41 suricata-homenet.yaml
> > -rw-r--r--  1 nobody nobody   98 Apr 10 14:40 suricata-http-
> > ports.yaml
> > -rw-r--r--  1 nobody nobody   95 Apr 10 14:54 suricata-static-
> > included-rulefiles.yaml
> > -rw-r--r--  1 nobody nobody   76 Apr 10 22:47 suricata-urlhaus-
> > used-
> > rulefiles.yaml
> > -rw-r--r--  1 nobody nobody  214 Apr 10 14:54 suricata-used-
> > providers.yaml
> > 
> > Three of the files are owned root:root while all the others are
> > nobody:nobody
> > 
> > 
> > The above was with extracting and applying the updated tar file on
> > top of IPFire after running the last version.
> > 
> > I will do a fresh clone of my IPFire vm and then repeat the tar
> > extraction and convert and see if that gives any difference.
> > 
> > 
> > Regards,
> > 
> > Adolf
> > 
> > On 10/04/2021 20:25, Stefan Schantl wrote:
> > > Hello list followers,
> > > 
> > > after getting a lot of feedback and bug reports I'm happy to
> > > announce the third test version for the new IDS system.
> > > 
> > > https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-003.tar.gz
> > > 
> > > If you just join testing, please omit the installation
> > > instructions
> > > from the initial Mail from this list.
> > > 
> > > The converter script now works as expected and runs very smooth.
> > > 
> > > As usual please post your feedback and opinions to this list and
> > > any
> > > remain bugs to our bugtracker. (https://bugzilla.ipfire.org)
> > > 
> > > A big thanks in advance,
> > > 
> > > -Stefan
> > > 

Re: IDS with support for multiple ruleset providers

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 4273 bytes --]

Hi Stefan,

I have installed the new version from scratch in my ipfire vm testbed. I followed "all" the instructions this time :-)

I was able to add additional providers and then go and select the rules I wanted and had no problems at all.

Looks like all fixed. I will do further evaluation of it over the next few days and let you know how things go for me.

Regards,

Adolf.

On 11/04/2021 10:46, Stefan Schantl wrote:
> Hello again,
> 
> I've tested and uploaded the fourth test verstion.
> 
> https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-004.tar.gz
> 
> This time the ownership of all files are correct at my test system.
> 
> (Tested with ruleset changes and without)
> 
> Best regards,
> 
> -Stefan
> 
>> Best regards,
>>
>> -Stefan
>>
>>> Hi Stefan,
>>>
>>> I copied the new tarfile to my ipfire vm testbed machine and
>>> extracted it and ran the converter script. No errors. I then used
>>> the
>>> wui page to add a new provider to the list then selected to
>>> customize
>>> the rules and ticked the box for the added rules. Then I pressed
>>> apply and got a blank white screen again.
>>>
>>>
>>> The error log has the following:-
>>>
>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>> 288.
>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>> 288.
>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>> 288.
>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>> 288.
>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>> 288.
>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>> 288.
>>> Could not open /var/ipfire/suricata/oinkmaster-provider-
>>> includes.conf. Permission denied
>>>
>>>
>>> ls- hal of /var/ipfire/suricata shows the following
>>>
>>> drwxr-xr-x  2 nobody nobody 4.0K Apr 10 22:47 .
>>> drwxr-xr-x 49 root   root   4.0K Apr  5 08:20 ..
>>> -rw-r--r--  1 nobody nobody    0 Dec 14 19:05 ignored
>>> -rw-r--r--  1 root   root    21K Apr  1 20:00 oinkmaster.conf
>>> -rw-r--r--  1 nobody nobody   61 Apr 10 14:40 oinkmaster-modify-
>>> sids.conf
>>> -rw-r--r--  1 root   root      0 Apr 10 14:54 oinkmaster-provider-
>>> includes.conf
>>> -rw-r--r--  1 nobody nobody   55 Apr 10 22:47 providers-settings
>>> -rw-r--r--  1 root   root   6.0K Apr  5 07:13 ruleset-sources
>>> -rw-r--r--  1 nobody nobody  102 Apr 10 14:54 settings
>>> -rw-r--r--  1 nobody nobody  140 Apr 10 22:41 suricata-dns-
>>> servers.yaml
>>> -rw-r--r--  1 nobody nobody  125 Apr 10 14:54 suricata-emerging-
>>> used-
>>> rulefiles.yaml
>>> -rw-r--r--  1 nobody nobody  159 Apr 10 22:41 suricata-homenet.yaml
>>> -rw-r--r--  1 nobody nobody   98 Apr 10 14:40 suricata-http-
>>> ports.yaml
>>> -rw-r--r--  1 nobody nobody   95 Apr 10 14:54 suricata-static-
>>> included-rulefiles.yaml
>>> -rw-r--r--  1 nobody nobody   76 Apr 10 22:47 suricata-urlhaus-
>>> used-
>>> rulefiles.yaml
>>> -rw-r--r--  1 nobody nobody  214 Apr 10 14:54 suricata-used-
>>> providers.yaml
>>>
>>> Three of the files are owned root:root while all the others are
>>> nobody:nobody
>>>
>>>
>>> The above was with extracting and applying the updated tar file on
>>> top of IPFire after running the last version.
>>>
>>> I will do a fresh clone of my IPFire vm and then repeat the tar
>>> extraction and convert and see if that gives any difference.
>>>
>>>
>>> Regards,
>>>
>>> Adolf
>>>
>>> On 10/04/2021 20:25, Stefan Schantl wrote:
>>>> Hello list followers,
>>>>
>>>> after getting a lot of feedback and bug reports I'm happy to
>>>> announce the third test version for the new IDS system.
>>>>
>>>> https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-003.tar.gz
>>>>
>>>> If you just join testing, please omit the installation
>>>> instructions
>>>> from the initial Mail from this list.
>>>>
>>>> The converter script now works as expected and runs very smooth.
>>>>
>>>> As usual please post your feedback and opinions to this list and
>>>> any
>>>> remain bugs to our bugtracker. (https://bugzilla.ipfire.org)
>>>>
>>>> A big thanks in advance,
>>>>
>>>> -Stefan
>>>>
> 

Re: IDS with support for multiple ruleset providers

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 6509 bytes --]

Hi Stefan,

I did a bit more testing.

I added the snort community rules set. I then went to customise and left the snort rules unchecked then pressed apply.

I then disabled the snort rules from the main page and on the customise page the snort rules were no longer showing.

I then enabled the snort rules on the first page and then went to customise but the snort rules still were not showing.

I deleted the snort ruleset provider on the first page and then added them back and now the snort ruleset was shown again on the customise page.

I then checked the snort ruleset and applied it and then entered customise again and unchecked the snort ruleset and applied it. When I went back into customise the snort ruleset was checked again. So once checked I could not uncheck it and keep it that why by pressing apply.

I then deleted the snort ruleset provider from the first page. Then the ruleset was gone from the customise page.

Then I added the snort ruleset provider back in but then got an error message saying that the snort ruleset provider was already selected. I then pressed back and came back to the main page with no snort ruleset provider but also with the page  only showing down to the Ruleset Settings table. There was nothing else after that.

The httpd/error_log showed the following

Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
Can't call method "mtime" on an undefined value at /var/ipfire/ids-functions.pl line 1512

Reloading the IPFire browser page and going back to the IDS main page gives the same result with the additional two lines in the log

Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
Can't call method "mtime" on an undefined value at /var/ipfire/ids-functions.pl line 1512.


Sorry for breaking it again. If any of my steps are not clear let me know and I will clarify where necessary.


Regards,

Adolf.


On 11/04/2021 11:49, Adolf Belka wrote:
> Hi Stefan,
>
> I have installed the new version from scratch in my ipfire vm testbed. I followed "all" the instructions this time :-)
>
> I was able to add additional providers and then go and select the rules I wanted and had no problems at all.
>
> Looks like all fixed. I will do further evaluation of it over the next few days and let you know how things go for me.
>
> Regards,
>
> Adolf.
>
> On 11/04/2021 10:46, Stefan Schantl wrote:
>> Hello again,
>>
>> I've tested and uploaded the fourth test verstion.
>>
>> https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-004.tar.gz
>>
>> This time the ownership of all files are correct at my test system.
>>
>> (Tested with ruleset changes and without)
>>
>> Best regards,
>>
>> -Stefan
>>
>>> Best regards,
>>>
>>> -Stefan
>>>
>>>> Hi Stefan,
>>>>
>>>> I copied the new tarfile to my ipfire vm testbed machine and
>>>> extracted it and ran the converter script. No errors. I then used
>>>> the
>>>> wui page to add a new provider to the list then selected to
>>>> customize
>>>> the rules and ticked the box for the added rules. Then I pressed
>>>> apply and got a blank white screen again.
>>>>
>>>>
>>>> The error log has the following:-
>>>>
>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>>> 288.
>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>>> 288.
>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>>> 288.
>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>>> 288.
>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>>> 288.
>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>>> 288.
>>>> Could not open /var/ipfire/suricata/oinkmaster-provider-
>>>> includes.conf. Permission denied
>>>>
>>>>
>>>> ls- hal of /var/ipfire/suricata shows the following
>>>>
>>>> drwxr-xr-x  2 nobody nobody 4.0K Apr 10 22:47 .
>>>> drwxr-xr-x 49 root   root   4.0K Apr  5 08:20 ..
>>>> -rw-r--r--  1 nobody nobody    0 Dec 14 19:05 ignored
>>>> -rw-r--r--  1 root   root    21K Apr  1 20:00 oinkmaster.conf
>>>> -rw-r--r--  1 nobody nobody   61 Apr 10 14:40 oinkmaster-modify-
>>>> sids.conf
>>>> -rw-r--r--  1 root   root      0 Apr 10 14:54 oinkmaster-provider-
>>>> includes.conf
>>>> -rw-r--r--  1 nobody nobody   55 Apr 10 22:47 providers-settings
>>>> -rw-r--r--  1 root   root   6.0K Apr  5 07:13 ruleset-sources
>>>> -rw-r--r--  1 nobody nobody  102 Apr 10 14:54 settings
>>>> -rw-r--r--  1 nobody nobody  140 Apr 10 22:41 suricata-dns-
>>>> servers.yaml
>>>> -rw-r--r--  1 nobody nobody  125 Apr 10 14:54 suricata-emerging-
>>>> used-
>>>> rulefiles.yaml
>>>> -rw-r--r--  1 nobody nobody  159 Apr 10 22:41 suricata-homenet.yaml
>>>> -rw-r--r--  1 nobody nobody   98 Apr 10 14:40 suricata-http-
>>>> ports.yaml
>>>> -rw-r--r--  1 nobody nobody   95 Apr 10 14:54 suricata-static-
>>>> included-rulefiles.yaml
>>>> -rw-r--r--  1 nobody nobody   76 Apr 10 22:47 suricata-urlhaus-
>>>> used-
>>>> rulefiles.yaml
>>>> -rw-r--r--  1 nobody nobody  214 Apr 10 14:54 suricata-used-
>>>> providers.yaml
>>>>
>>>> Three of the files are owned root:root while all the others are
>>>> nobody:nobody
>>>>
>>>>
>>>> The above was with extracting and applying the updated tar file on
>>>> top of IPFire after running the last version.
>>>>
>>>> I will do a fresh clone of my IPFire vm and then repeat the tar
>>>> extraction and convert and see if that gives any difference.
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Adolf
>>>>
>>>> On 10/04/2021 20:25, Stefan Schantl wrote:
>>>>> Hello list followers,
>>>>>
>>>>> after getting a lot of feedback and bug reports I'm happy to
>>>>> announce the third test version for the new IDS system.
>>>>>
>>>>> https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-003.tar.gz
>>>>>
>>>>> If you just join testing, please omit the installation
>>>>> instructions
>>>>> from the initial Mail from this list.
>>>>>
>>>>> The converter script now works as expected and runs very smooth.
>>>>>
>>>>> As usual please post your feedback and opinions to this list and
>>>>> any
>>>>> remain bugs to our bugtracker. (https://bugzilla.ipfire.org)
>>>>>
>>>>> A big thanks in advance,
>>>>>
>>>>> -Stefan
>>>>>
>>

Re: IDS with support for multiple ruleset providers

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 6892 bytes --]

Hello,

> On 11 Apr 2021, at 11:18, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Stefan,
> 
> I did a bit more testing.
> 
> I added the snort community rules set. I then went to customise and left the snort rules unchecked then pressed apply.
> 
> I then disabled the snort rules from the main page and on the customise page the snort rules were no longer showing.
> 
> I then enabled the snort rules on the first page and then went to customise but the snort rules still were not showing.
> 
> I deleted the snort ruleset provider on the first page and then added them back and now the snort ruleset was shown again on the customise page.
> 
> I then checked the snort ruleset and applied it and then entered customise again and unchecked the snort ruleset and applied it. When I went back into customise the snort ruleset was checked again. So once checked I could not uncheck it and keep it that why by pressing apply.
> 
> I then deleted the snort ruleset provider from the first page. Then the ruleset was gone from the customise page.
> 
> Then I added the snort ruleset provider back in but then got an error message saying that the snort ruleset provider was already selected. I then pressed back and came back to the main page with no snort ruleset provider but also with the page  only showing down to the Ruleset Settings table. There was nothing else after that.
> 
> The httpd/error_log showed the following
> 
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
> Can't call method "mtime" on an undefined value at /var/ipfire/ids-functions.pl line 1512
> 
> Reloading the IPFire browser page and going back to the IDS main page gives the same result with the additional two lines in the log
> 
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line 288.
> Can't call method "mtime" on an undefined value at /var/ipfire/ids-functions.pl line 1512.
> 
> 
> Sorry for breaking it again. If any of my steps are not clear let me know and I will clarify where necessary.

This is absolutely the idea here. Clicking all the buttons as fast as we can until it breaks :)

-Michael

> Regards,
> 
> Adolf.
> 
> 
> On 11/04/2021 11:49, Adolf Belka wrote:
>> Hi Stefan,
>> 
>> I have installed the new version from scratch in my ipfire vm testbed. I followed "all" the instructions this time :-)
>> 
>> I was able to add additional providers and then go and select the rules I wanted and had no problems at all.
>> 
>> Looks like all fixed. I will do further evaluation of it over the next few days and let you know how things go for me.
>> 
>> Regards,
>> 
>> Adolf.
>> 
>> On 11/04/2021 10:46, Stefan Schantl wrote:
>>> Hello again,
>>> 
>>> I've tested and uploaded the fourth test verstion.
>>> 
>>> https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-004.tar.gz
>>> 
>>> This time the ownership of all files are correct at my test system.
>>> 
>>> (Tested with ruleset changes and without)
>>> 
>>> Best regards,
>>> 
>>> -Stefan
>>> 
>>>> Best regards,
>>>> 
>>>> -Stefan
>>>> 
>>>>> Hi Stefan,
>>>>> 
>>>>> I copied the new tarfile to my ipfire vm testbed machine and
>>>>> extracted it and ran the converter script. No errors. I then used
>>>>> the
>>>>> wui page to add a new provider to the list then selected to
>>>>> customize
>>>>> the rules and ticked the box for the added rules. Then I pressed
>>>>> apply and got a blank white screen again.
>>>>> 
>>>>> 
>>>>> The error log has the following:-
>>>>> 
>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>>>> 288.
>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>>>> 288.
>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>>>> 288.
>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>>>> 288.
>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>>>> 288.
>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>>>> 288.
>>>>> Could not open /var/ipfire/suricata/oinkmaster-provider-
>>>>> includes.conf. Permission denied
>>>>> 
>>>>> 
>>>>> ls- hal of /var/ipfire/suricata shows the following
>>>>> 
>>>>> drwxr-xr-x  2 nobody nobody 4.0K Apr 10 22:47 .
>>>>> drwxr-xr-x 49 root   root   4.0K Apr  5 08:20 ..
>>>>> -rw-r--r--  1 nobody nobody    0 Dec 14 19:05 ignored
>>>>> -rw-r--r--  1 root   root    21K Apr  1 20:00 oinkmaster.conf
>>>>> -rw-r--r--  1 nobody nobody   61 Apr 10 14:40 oinkmaster-modify-
>>>>> sids.conf
>>>>> -rw-r--r--  1 root   root      0 Apr 10 14:54 oinkmaster-provider-
>>>>> includes.conf
>>>>> -rw-r--r--  1 nobody nobody   55 Apr 10 22:47 providers-settings
>>>>> -rw-r--r--  1 root   root   6.0K Apr  5 07:13 ruleset-sources
>>>>> -rw-r--r--  1 nobody nobody  102 Apr 10 14:54 settings
>>>>> -rw-r--r--  1 nobody nobody  140 Apr 10 22:41 suricata-dns-
>>>>> servers.yaml
>>>>> -rw-r--r--  1 nobody nobody  125 Apr 10 14:54 suricata-emerging-
>>>>> used-
>>>>> rulefiles.yaml
>>>>> -rw-r--r--  1 nobody nobody  159 Apr 10 22:41 suricata-homenet.yaml
>>>>> -rw-r--r--  1 nobody nobody   98 Apr 10 14:40 suricata-http-
>>>>> ports.yaml
>>>>> -rw-r--r--  1 nobody nobody   95 Apr 10 14:54 suricata-static-
>>>>> included-rulefiles.yaml
>>>>> -rw-r--r--  1 nobody nobody   76 Apr 10 22:47 suricata-urlhaus-
>>>>> used-
>>>>> rulefiles.yaml
>>>>> -rw-r--r--  1 nobody nobody  214 Apr 10 14:54 suricata-used-
>>>>> providers.yaml
>>>>> 
>>>>> Three of the files are owned root:root while all the others are
>>>>> nobody:nobody
>>>>> 
>>>>> 
>>>>> The above was with extracting and applying the updated tar file on
>>>>> top of IPFire after running the last version.
>>>>> 
>>>>> I will do a fresh clone of my IPFire vm and then repeat the tar
>>>>> extraction and convert and see if that gives any difference.
>>>>> 
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> Adolf
>>>>> 
>>>>> On 10/04/2021 20:25, Stefan Schantl wrote:
>>>>>> Hello list followers,
>>>>>> 
>>>>>> after getting a lot of feedback and bug reports I'm happy to
>>>>>> announce the third test version for the new IDS system.
>>>>>> 
>>>>>> https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-003.tar.gz
>>>>>> 
>>>>>> If you just join testing, please omit the installation
>>>>>> instructions
>>>>>> from the initial Mail from this list.
>>>>>> 
>>>>>> The converter script now works as expected and runs very smooth.
>>>>>> 
>>>>>> As usual please post your feedback and opinions to this list and
>>>>>> any
>>>>>> remain bugs to our bugtracker. (https://bugzilla.ipfire.org)
>>>>>> 
>>>>>> A big thanks in advance,
>>>>>> 
>>>>>> -Stefan
>>>>>> 
>>> 

Re: IDS with support for multiple ruleset providers

[Stefan Schantl]

[-- Attachment #1: Type: text/plain, Size: 7974 bytes --]

Hello Adolf,

thanks you very much for your huge effort in testing this and providing
this very detailed feedback.

While reading through your single steps it feels sometimes near to get
a knot inside my brain....

> Hi Stefan,
> 
> I did a bit more testing.
> 
> I added the snort community rules set. I then went to customise and
> left the snort rules unchecked then pressed apply.
> 
> I then disabled the snort rules from the main page and on the
> customise page the snort rules were no longer showing.

Workes as designed.

> 
> I then enabled the snort rules on the first page and then went to
> customise but the snort rules still were not showing.

Very good catch - Fixed.

> 
> I deleted the snort ruleset provider on the first page and then added
> them back and now the snort ruleset was shown again on the customise
> page.

OK.

> 
> I then checked the snort ruleset and applied it and then entered
> customise again and unchecked the snort ruleset and applied it. When
> I went back into customise the snort ruleset was checked again. So
> once checked I could not uncheck it and keep it that why by pressing
> apply.
> 

Confirmed. Thanks for finding this.

> I then deleted the snort ruleset provider from the first page. Then
> the ruleset was gone from the customise page.
> 
> Then I added the snort ruleset provider back in but then got an error
> message saying that the snort ruleset provider was already selected.
> I then pressed back and came back to the main page with no snort
> ruleset provider but also with the page  only showing down to the
> Ruleset Settings table. There was nothing else after that.
> 
> The httpd/error_log showed the following
> 
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> 288.
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> 288.
> Can't call method "mtime" on an undefined value at /var/ipfire/ids-
> functions.pl line 1512
> 
> Reloading the IPFire browser page and going back to the IDS main page
> gives the same result with the additional two lines in the log
> 
> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> 288.
> Can't call method "mtime" on an undefined value at /var/ipfire/ids-
> functions.pl line 1512.
> 

Sorry I'm unable to reproduce this - maybe a download error?

> 
> Sorry for breaking it again. If any of my steps are not clear let me
> know and I will clarify where necessary.

Hey, this is why we do testing - each found bug until release is a good
bug!

> 
> 
> Regards,
> 
> Adolf.
> 
> 
> On 11/04/2021 11:49, Adolf Belka wrote:
> > Hi Stefan,
> > 
> > I have installed the new version from scratch in my ipfire vm
> > testbed. I followed "all" the instructions this time :-)
> > 
> > I was able to add additional providers and then go and select the
> > rules I wanted and had no problems at all.
> > 
> > Looks like all fixed. I will do further evaluation of it over the
> > next few days and let you know how things go for me.
> > 
> > Regards,
> > 
> > Adolf.
> > 
> > On 11/04/2021 10:46, Stefan Schantl wrote:
> > > Hello again,
> > > 
> > > I've tested and uploaded the fourth test verstion.
> > > 
> > > https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-004.tar.gz
> > > 
> > > This time the ownership of all files are correct at my test
> > > system.
> > > 
> > > (Tested with ruleset changes and without)
> > > 
> > > Best regards,
> > > 
> > > -Stefan
> > > 
> > > > Best regards,
> > > > 
> > > > -Stefan
> > > > 
> > > > > Hi Stefan,
> > > > > 
> > > > > I copied the new tarfile to my ipfire vm testbed machine and
> > > > > extracted it and ran the converter script. No errors. I then
> > > > > used
> > > > > the
> > > > > wui page to add a new provider to the list then selected to
> > > > > customize
> > > > > the rules and ticked the box for the added rules. Then I
> > > > > pressed
> > > > > apply and got a blank white screen again.
> > > > > 
> > > > > 
> > > > > The error log has the following:-
> > > > > 
> > > > > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi 
> > > > > line
> > > > > 288.
> > > > > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi 
> > > > > line
> > > > > 288.
> > > > > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi 
> > > > > line
> > > > > 288.
> > > > > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi 
> > > > > line
> > > > > 288.
> > > > > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi 
> > > > > line
> > > > > 288.
> > > > > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi 
> > > > > line
> > > > > 288.
> > > > > Could not open /var/ipfire/suricata/oinkmaster-provider-
> > > > > includes.conf. Permission denied
> > > > > 
> > > > > 
> > > > > ls- hal of /var/ipfire/suricata shows the following
> > > > > 
> > > > > drwxr-xr-x  2 nobody nobody 4.0K Apr 10 22:47 .
> > > > > drwxr-xr-x 49 root   root   4.0K Apr  5 08:20 ..
> > > > > -rw-r--r--  1 nobody nobody    0 Dec 14 19:05 ignored
> > > > > -rw-r--r--  1 root   root    21K Apr  1 20:00 oinkmaster.conf
> > > > > -rw-r--r--  1 nobody nobody   61 Apr 10 14:40 oinkmaster-
> > > > > modify-
> > > > > sids.conf
> > > > > -rw-r--r--  1 root   root      0 Apr 10 14:54 oinkmaster-
> > > > > provider-
> > > > > includes.conf
> > > > > -rw-r--r--  1 nobody nobody   55 Apr 10 22:47 providers-
> > > > > settings
> > > > > -rw-r--r--  1 root   root   6.0K Apr  5 07:13 ruleset-sources
> > > > > -rw-r--r--  1 nobody nobody  102 Apr 10 14:54 settings
> > > > > -rw-r--r--  1 nobody nobody  140 Apr 10 22:41 suricata-dns-
> > > > > servers.yaml
> > > > > -rw-r--r--  1 nobody nobody  125 Apr 10 14:54 suricata-
> > > > > emerging-
> > > > > used-
> > > > > rulefiles.yaml
> > > > > -rw-r--r--  1 nobody nobody  159 Apr 10 22:41 suricata-
> > > > > homenet.yaml
> > > > > -rw-r--r--  1 nobody nobody   98 Apr 10 14:40 suricata-http-
> > > > > ports.yaml
> > > > > -rw-r--r--  1 nobody nobody   95 Apr 10 14:54 suricata-
> > > > > static-
> > > > > included-rulefiles.yaml
> > > > > -rw-r--r--  1 nobody nobody   76 Apr 10 22:47 suricata-
> > > > > urlhaus-
> > > > > used-
> > > > > rulefiles.yaml
> > > > > -rw-r--r--  1 nobody nobody  214 Apr 10 14:54 suricata-used-
> > > > > providers.yaml
> > > > > 
> > > > > Three of the files are owned root:root while all the others
> > > > > are
> > > > > nobody:nobody
> > > > > 
> > > > > 
> > > > > The above was with extracting and applying the updated tar
> > > > > file on
> > > > > top of IPFire after running the last version.
> > > > > 
> > > > > I will do a fresh clone of my IPFire vm and then repeat the
> > > > > tar
> > > > > extraction and convert and see if that gives any difference.
> > > > > 
> > > > > 
> > > > > Regards,
> > > > > 
> > > > > Adolf
> > > > > 
> > > > > On 10/04/2021 20:25, Stefan Schantl wrote:
> > > > > > Hello list followers,
> > > > > > 
> > > > > > after getting a lot of feedback and bug reports I'm happy
> > > > > > to
> > > > > > announce the third test version for the new IDS system.
> > > > > > 
> > > > > > https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-003.tar.gz
> > > > > > 
> > > > > > If you just join testing, please omit the installation
> > > > > > instructions
> > > > > > from the initial Mail from this list.
> > > > > > 
> > > > > > The converter script now works as expected and runs very
> > > > > > smooth.
> > > > > > 
> > > > > > As usual please post your feedback and opinions to this
> > > > > > list and
> > > > > > any
> > > > > > remain bugs to our bugtracker. (https://bugzilla.ipfire.org
> > > > > > )
> > > > > > 
> > > > > > A big thanks in advance,
> > > > > > 
> > > > > > -Stefan
> > > > > > 

Re: IDS with support for multiple ruleset providers

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 8142 bytes --]



> On 13 Apr 2021, at 19:57, Stefan Schantl <stefan.schantl(a)ipfire.org> wrote:
> 
> Hello Adolf,
> 
> thanks you very much for your huge effort in testing this and providing
> this very detailed feedback.
> 
> While reading through your single steps it feels sometimes near to get
> a knot inside my brain....
> 
>> Hi Stefan,
>> 
>> I did a bit more testing.
>> 
>> I added the snort community rules set. I then went to customise and
>> left the snort rules unchecked then pressed apply.
>> 
>> I then disabled the snort rules from the main page and on the
>> customise page the snort rules were no longer showing.
> 
> Workes as designed.
> 
>> 
>> I then enabled the snort rules on the first page and then went to
>> customise but the snort rules still were not showing.
> 
> Very good catch - Fixed.
> 
>> 
>> I deleted the snort ruleset provider on the first page and then added
>> them back and now the snort ruleset was shown again on the customise
>> page.
> 
> OK.
> 
>> 
>> I then checked the snort ruleset and applied it and then entered
>> customise again and unchecked the snort ruleset and applied it. When
>> I went back into customise the snort ruleset was checked again. So
>> once checked I could not uncheck it and keep it that why by pressing
>> apply.
>> 
> 
> Confirmed. Thanks for finding this.
> 
>> I then deleted the snort ruleset provider from the first page. Then
>> the ruleset was gone from the customise page.
>> 
>> Then I added the snort ruleset provider back in but then got an error
>> message saying that the snort ruleset provider was already selected.
>> I then pressed back and came back to the main page with no snort
>> ruleset provider but also with the page  only showing down to the
>> Ruleset Settings table. There was nothing else after that.
>> 
>> The httpd/error_log showed the following
>> 
>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>> 288.
>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>> 288.
>> Can't call method "mtime" on an undefined value at /var/ipfire/ids-
>> functions.pl line 1512
>> 
>> Reloading the IPFire browser page and going back to the IDS main page
>> gives the same result with the additional two lines in the log
>> 
>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>> 288.
>> Can't call method "mtime" on an undefined value at /var/ipfire/ids-
>> functions.pl line 1512.
>> 
> 
> Sorry I'm unable to reproduce this - maybe a download error?

Might be. You do not check if stat() was successful and continue working with the result:

  https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=blob;f=config/cfgroot/ids-functions.pl;h=7e7ad46b53ee1481d1c56f436ff0ee2636e767ce;hb=de30329f3b089302969d5f79709855b57605df57#l1498

Just check if stat() returned something useful before continuing.

>> 
>> Sorry for breaking it again. If any of my steps are not clear let me
>> know and I will clarify where necessary.
> 
> Hey, this is why we do testing - each found bug until release is a good
> bug!

:)

-Michael

> 
>> 
>> 
>> Regards,
>> 
>> Adolf.
>> 
>> 
>> On 11/04/2021 11:49, Adolf Belka wrote:
>>> Hi Stefan,
>>> 
>>> I have installed the new version from scratch in my ipfire vm
>>> testbed. I followed "all" the instructions this time :-)
>>> 
>>> I was able to add additional providers and then go and select the
>>> rules I wanted and had no problems at all.
>>> 
>>> Looks like all fixed. I will do further evaluation of it over the
>>> next few days and let you know how things go for me.
>>> 
>>> Regards,
>>> 
>>> Adolf.
>>> 
>>> On 11/04/2021 10:46, Stefan Schantl wrote:
>>>> Hello again,
>>>> 
>>>> I've tested and uploaded the fourth test verstion.
>>>> 
>>>> https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-004.tar.gz
>>>> 
>>>> This time the ownership of all files are correct at my test
>>>> system.
>>>> 
>>>> (Tested with ruleset changes and without)
>>>> 
>>>> Best regards,
>>>> 
>>>> -Stefan
>>>> 
>>>>> Best regards,
>>>>> 
>>>>> -Stefan
>>>>> 
>>>>>> Hi Stefan,
>>>>>> 
>>>>>> I copied the new tarfile to my ipfire vm testbed machine and
>>>>>> extracted it and ran the converter script. No errors. I then
>>>>>> used
>>>>>> the
>>>>>> wui page to add a new provider to the list then selected to
>>>>>> customize
>>>>>> the rules and ticked the box for the added rules. Then I
>>>>>> pressed
>>>>>> apply and got a blank white screen again.
>>>>>> 
>>>>>> 
>>>>>> The error log has the following:-
>>>>>> 
>>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi 
>>>>>> line
>>>>>> 288.
>>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi 
>>>>>> line
>>>>>> 288.
>>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi 
>>>>>> line
>>>>>> 288.
>>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi 
>>>>>> line
>>>>>> 288.
>>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi 
>>>>>> line
>>>>>> 288.
>>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi 
>>>>>> line
>>>>>> 288.
>>>>>> Could not open /var/ipfire/suricata/oinkmaster-provider-
>>>>>> includes.conf. Permission denied
>>>>>> 
>>>>>> 
>>>>>> ls- hal of /var/ipfire/suricata shows the following
>>>>>> 
>>>>>> drwxr-xr-x  2 nobody nobody 4.0K Apr 10 22:47 .
>>>>>> drwxr-xr-x 49 root   root   4.0K Apr  5 08:20 ..
>>>>>> -rw-r--r--  1 nobody nobody    0 Dec 14 19:05 ignored
>>>>>> -rw-r--r--  1 root   root    21K Apr  1 20:00 oinkmaster.conf
>>>>>> -rw-r--r--  1 nobody nobody   61 Apr 10 14:40 oinkmaster-
>>>>>> modify-
>>>>>> sids.conf
>>>>>> -rw-r--r--  1 root   root      0 Apr 10 14:54 oinkmaster-
>>>>>> provider-
>>>>>> includes.conf
>>>>>> -rw-r--r--  1 nobody nobody   55 Apr 10 22:47 providers-
>>>>>> settings
>>>>>> -rw-r--r--  1 root   root   6.0K Apr  5 07:13 ruleset-sources
>>>>>> -rw-r--r--  1 nobody nobody  102 Apr 10 14:54 settings
>>>>>> -rw-r--r--  1 nobody nobody  140 Apr 10 22:41 suricata-dns-
>>>>>> servers.yaml
>>>>>> -rw-r--r--  1 nobody nobody  125 Apr 10 14:54 suricata-
>>>>>> emerging-
>>>>>> used-
>>>>>> rulefiles.yaml
>>>>>> -rw-r--r--  1 nobody nobody  159 Apr 10 22:41 suricata-
>>>>>> homenet.yaml
>>>>>> -rw-r--r--  1 nobody nobody   98 Apr 10 14:40 suricata-http-
>>>>>> ports.yaml
>>>>>> -rw-r--r--  1 nobody nobody   95 Apr 10 14:54 suricata-
>>>>>> static-
>>>>>> included-rulefiles.yaml
>>>>>> -rw-r--r--  1 nobody nobody   76 Apr 10 22:47 suricata-
>>>>>> urlhaus-
>>>>>> used-
>>>>>> rulefiles.yaml
>>>>>> -rw-r--r--  1 nobody nobody  214 Apr 10 14:54 suricata-used-
>>>>>> providers.yaml
>>>>>> 
>>>>>> Three of the files are owned root:root while all the others
>>>>>> are
>>>>>> nobody:nobody
>>>>>> 
>>>>>> 
>>>>>> The above was with extracting and applying the updated tar
>>>>>> file on
>>>>>> top of IPFire after running the last version.
>>>>>> 
>>>>>> I will do a fresh clone of my IPFire vm and then repeat the
>>>>>> tar
>>>>>> extraction and convert and see if that gives any difference.
>>>>>> 
>>>>>> 
>>>>>> Regards,
>>>>>> 
>>>>>> Adolf
>>>>>> 
>>>>>> On 10/04/2021 20:25, Stefan Schantl wrote:
>>>>>>> Hello list followers,
>>>>>>> 
>>>>>>> after getting a lot of feedback and bug reports I'm happy
>>>>>>> to
>>>>>>> announce the third test version for the new IDS system.
>>>>>>> 
>>>>>>> https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-003.tar.gz
>>>>>>> 
>>>>>>> If you just join testing, please omit the installation
>>>>>>> instructions
>>>>>>> from the initial Mail from this list.
>>>>>>> 
>>>>>>> The converter script now works as expected and runs very
>>>>>>> smooth.
>>>>>>> 
>>>>>>> As usual please post your feedback and opinions to this
>>>>>>> list and
>>>>>>> any
>>>>>>> remain bugs to our bugtracker. (https://bugzilla.ipfire.org
>>>>>>> )
>>>>>>> 
>>>>>>> A big thanks in advance,
>>>>>>> 
>>>>>>> -Stefan

Re: IDS with support for multiple ruleset providers

[Stefan Schantl]

[-- Attachment #1: Type: text/plain, Size: 10425 bytes --]

Hello Michael,

> > On 13 Apr 2021, at 19:57, Stefan Schantl <stefan.schantl(a)ipfire.org
> > > wrote:
> > 
> > Hello Adolf,
> > 
> > thanks you very much for your huge effort in testing this and
> > providing
> > this very detailed feedback.
> > 
> > While reading through your single steps it feels sometimes near to
> > get
> > a knot inside my brain....
> > 
> > > Hi Stefan,
> > > 
> > > I did a bit more testing.
> > > 
> > > I added the snort community rules set. I then went to customise
> > > and
> > > left the snort rules unchecked then pressed apply.
> > > 
> > > I then disabled the snort rules from the main page and on the
> > > customise page the snort rules were no longer showing.
> > 
> > Workes as designed.
> > 
> > > I then enabled the snort rules on the first page and then went to
> > > customise but the snort rules still were not showing.
> > 
> > Very good catch - Fixed.
> > 
> > > I deleted the snort ruleset provider on the first page and then
> > > added
> > > them back and now the snort ruleset was shown again on the
> > > customise
> > > page.
> > 
> > OK.
> > 
> > > I then checked the snort ruleset and applied it and then entered
> > > customise again and unchecked the snort ruleset and applied it.
> > > When
> > > I went back into customise the snort ruleset was checked again.
> > > So
> > > once checked I could not uncheck it and keep it that why by
> > > pressing
> > > apply.
> > > 
> > 
> > Confirmed. Thanks for finding this.
> > 
> > > I then deleted the snort ruleset provider from the first page.
> > > Then
> > > the ruleset was gone from the customise page.
> > > 
> > > Then I added the snort ruleset provider back in but then got an
> > > error
> > > message saying that the snort ruleset provider was already
> > > selected.
> > > I then pressed back and came back to the main page with no snort
> > > ruleset provider but also with the page  only showing down to the
> > > Ruleset Settings table. There was nothing else after that.
> > > 
> > > The httpd/error_log showed the following
> > > 
> > > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi
> > > line
> > > 288.
> > > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi
> > > line
> > > 288.
> > > Can't call method "mtime" on an undefined value at
> > > /var/ipfire/ids-
> > > functions.pl line 1512
> > > 
> > > Reloading the IPFire browser page and going back to the IDS main
> > > page
> > > gives the same result with the additional two lines in the log
> > > 
> > > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi
> > > line
> > > 288.
> > > Can't call method "mtime" on an undefined value at
> > > /var/ipfire/ids-
> > > functions.pl line 1512.
> > > 
> > 
> > Sorry I'm unable to reproduce this - maybe a download error?
> 
> Might be. You do not check if stat() was successful and continue
> working with the result:
> 
>   
> https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=blob;f=config/cfgroot/ids-functions.pl;h=7e7ad46b53ee1481d1c56f436ff0ee2636e767ce;hb=de30329f3b089302969d5f79709855b57605df57#l1498
> 

The main problem was a bit deeper:

If the system is offline or for some other reason the ruleset of the
newly added provider could not be downloaded. There was no error
handling for this case.

I've fixed this with the following commit:

https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=7cf0ecadc14c2a8f6a711ff3ff3dfa2c0a516fb5


> Just check if stat() returned something useful before continuing.
> 

You are right, this also needs to be done.

I've created the following commit for that:

https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=e59b8178e0cb4098904a8c0f591639d92a1f625e

> > > Sorry for breaking it again. If any of my steps are not clear let
> > > me
> > > know and I will clarify where necessary.
> > 
> > Hey, this is why we do testing - each found bug until release is a
> > good
> > bug!
> 
> :)
> 
> -Michael
> 
> > > 
> > > Regards,
> > > 
> > > Adolf.
> > > 
> > > 
> > > On 11/04/2021 11:49, Adolf Belka wrote:
> > > > Hi Stefan,
> > > > 
> > > > I have installed the new version from scratch in my ipfire vm
> > > > testbed. I followed "all" the instructions this time :-)
> > > > 
> > > > I was able to add additional providers and then go and select
> > > > the
> > > > rules I wanted and had no problems at all.
> > > > 
> > > > Looks like all fixed. I will do further evaluation of it over
> > > > the
> > > > next few days and let you know how things go for me.
> > > > 
> > > > Regards,
> > > > 
> > > > Adolf.
> > > > 
> > > > On 11/04/2021 10:46, Stefan Schantl wrote:
> > > > > Hello again,
> > > > > 
> > > > > I've tested and uploaded the fourth test verstion.
> > > > > 
> > > > > https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-004.tar.gz
> > > > > 
> > > > > This time the ownership of all files are correct at my test
> > > > > system.
> > > > > 
> > > > > (Tested with ruleset changes and without)
> > > > > 
> > > > > Best regards,
> > > > > 
> > > > > -Stefan
> > > > > 
> > > > > > Best regards,
> > > > > > 
> > > > > > -Stefan
> > > > > > 
> > > > > > > Hi Stefan,
> > > > > > > 
> > > > > > > I copied the new tarfile to my ipfire vm testbed machine
> > > > > > > and
> > > > > > > extracted it and ran the converter script. No errors. I
> > > > > > > then
> > > > > > > used
> > > > > > > the
> > > > > > > wui page to add a new provider to the list then selected
> > > > > > > to
> > > > > > > customize
> > > > > > > the rules and ticked the box for the added rules. Then I
> > > > > > > pressed
> > > > > > > apply and got a blank white screen again.
> > > > > > > 
> > > > > > > 
> > > > > > > The error log has the following:-
> > > > > > > 
> > > > > > > Smartmatch is experimental at /srv/web/ipfire/cgi-
> > > > > > > bin/ids.cgi 
> > > > > > > line
> > > > > > > 288.
> > > > > > > Smartmatch is experimental at /srv/web/ipfire/cgi-
> > > > > > > bin/ids.cgi 
> > > > > > > line
> > > > > > > 288.
> > > > > > > Smartmatch is experimental at /srv/web/ipfire/cgi-
> > > > > > > bin/ids.cgi 
> > > > > > > line
> > > > > > > 288.
> > > > > > > Smartmatch is experimental at /srv/web/ipfire/cgi-
> > > > > > > bin/ids.cgi 
> > > > > > > line
> > > > > > > 288.
> > > > > > > Smartmatch is experimental at /srv/web/ipfire/cgi-
> > > > > > > bin/ids.cgi 
> > > > > > > line
> > > > > > > 288.
> > > > > > > Smartmatch is experimental at /srv/web/ipfire/cgi-
> > > > > > > bin/ids.cgi 
> > > > > > > line
> > > > > > > 288.
> > > > > > > Could not open /var/ipfire/suricata/oinkmaster-provider-
> > > > > > > includes.conf. Permission denied
> > > > > > > 
> > > > > > > 
> > > > > > > ls- hal of /var/ipfire/suricata shows the following
> > > > > > > 
> > > > > > > drwxr-xr-x  2 nobody nobody 4.0K Apr 10 22:47 .
> > > > > > > drwxr-xr-x 49 root   root   4.0K Apr  5 08:20 ..
> > > > > > > -rw-r--r--  1 nobody nobody    0 Dec 14 19:05 ignored
> > > > > > > -rw-r--r--  1 root   root    21K Apr  1 20:00
> > > > > > > oinkmaster.conf
> > > > > > > -rw-r--r--  1 nobody nobody   61 Apr 10 14:40 oinkmaster-
> > > > > > > modify-
> > > > > > > sids.conf
> > > > > > > -rw-r--r--  1 root   root      0 Apr 10 14:54 oinkmaster-
> > > > > > > provider-
> > > > > > > includes.conf
> > > > > > > -rw-r--r--  1 nobody nobody   55 Apr 10 22:47 providers-
> > > > > > > settings
> > > > > > > -rw-r--r--  1 root   root   6.0K Apr  5 07:13 ruleset-
> > > > > > > sources
> > > > > > > -rw-r--r--  1 nobody nobody  102 Apr 10 14:54 settings
> > > > > > > -rw-r--r--  1 nobody nobody  140 Apr 10 22:41 suricata-
> > > > > > > dns-
> > > > > > > servers.yaml
> > > > > > > -rw-r--r--  1 nobody nobody  125 Apr 10 14:54 suricata-
> > > > > > > emerging-
> > > > > > > used-
> > > > > > > rulefiles.yaml
> > > > > > > -rw-r--r--  1 nobody nobody  159 Apr 10 22:41 suricata-
> > > > > > > homenet.yaml
> > > > > > > -rw-r--r--  1 nobody nobody   98 Apr 10 14:40 suricata-
> > > > > > > http-
> > > > > > > ports.yaml
> > > > > > > -rw-r--r--  1 nobody nobody   95 Apr 10 14:54 suricata-
> > > > > > > static-
> > > > > > > included-rulefiles.yaml
> > > > > > > -rw-r--r--  1 nobody nobody   76 Apr 10 22:47 suricata-
> > > > > > > urlhaus-
> > > > > > > used-
> > > > > > > rulefiles.yaml
> > > > > > > -rw-r--r--  1 nobody nobody  214 Apr 10 14:54 suricata-
> > > > > > > used-
> > > > > > > providers.yaml
> > > > > > > 
> > > > > > > Three of the files are owned root:root while all the
> > > > > > > others
> > > > > > > are
> > > > > > > nobody:nobody
> > > > > > > 
> > > > > > > 
> > > > > > > The above was with extracting and applying the updated
> > > > > > > tar
> > > > > > > file on
> > > > > > > top of IPFire after running the last version.
> > > > > > > 
> > > > > > > I will do a fresh clone of my IPFire vm and then repeat
> > > > > > > the
> > > > > > > tar
> > > > > > > extraction and convert and see if that gives any
> > > > > > > difference.
> > > > > > > 
> > > > > > > 
> > > > > > > Regards,
> > > > > > > 
> > > > > > > Adolf
> > > > > > > 
> > > > > > > On 10/04/2021 20:25, Stefan Schantl wrote:
> > > > > > > > Hello list followers,
> > > > > > > > 
> > > > > > > > after getting a lot of feedback and bug reports I'm
> > > > > > > > happy
> > > > > > > > to
> > > > > > > > announce the third test version for the new IDS system.
> > > > > > > > 
> > > > > > > > https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-003.tar.gz
> > > > > > > > 
> > > > > > > > If you just join testing, please omit the installation
> > > > > > > > instructions
> > > > > > > > from the initial Mail from this list.
> > > > > > > > 
> > > > > > > > The converter script now works as expected and runs
> > > > > > > > very
> > > > > > > > smooth.
> > > > > > > > 
> > > > > > > > As usual please post your feedback and opinions to this
> > > > > > > > list and
> > > > > > > > any
> > > > > > > > remain bugs to our bugtracker. (
> > > > > > > > https://bugzilla.ipfire.org
> > > > > > > > )
> > > > > > > > 
> > > > > > > > A big thanks in advance,
> > > > > > > > 
> > > > > > > > -Stefan

Re: IDS with support for multiple ruleset providers

[Stefan Schantl]

[-- Attachment #1: Type: text/plain, Size: 9099 bytes --]

Am Dienstag, den 13.04.2021, 20:57 +0200 schrieb Stefan Schantl:
> Hello Adolf,
> 
> thanks you very much for your huge effort in testing this and
> providing
> this very detailed feedback.
> 
> While reading through your single steps it feels sometimes near to
> get
> a knot inside my brain....
> 
> > Hi Stefan,
> > 
> > I did a bit more testing.
> > 
> > I added the snort community rules set. I then went to customise and
> > left the snort rules unchecked then pressed apply.
> > 
> > I then disabled the snort rules from the main page and on the
> > customise page the snort rules were no longer showing.
> 
> Workes as designed.
> 
> > I then enabled the snort rules on the first page and then went to
> > customise but the snort rules still were not showing.
> 
> Very good catch - Fixed.
> 
> > I deleted the snort ruleset provider on the first page and then
> > added
> > them back and now the snort ruleset was shown again on the
> > customise
> > page.
> 
> OK.
> 
> > I then checked the snort ruleset and applied it and then entered
> > customise again and unchecked the snort ruleset and applied it.
> > When
> > I went back into customise the snort ruleset was checked again. So
> > once checked I could not uncheck it and keep it that why by
> > pressing
> > apply.
> > 
> 
> Confirmed. Thanks for finding this.

Fixed by commit:

https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=79cc92267f1811beab84ae190fc1c3724a67e5f4

> 
> > I then deleted the snort ruleset provider from the first page. Then
> > the ruleset was gone from the customise page.
> > 
> > Then I added the snort ruleset provider back in but then got an
> > error
> > message saying that the snort ruleset provider was already
> > selected.
> > I then pressed back and came back to the main page with no snort
> > ruleset provider but also with the page  only showing down to the
> > Ruleset Settings table. There was nothing else after that.
> > 
> > The httpd/error_log showed the following
> > 
> > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> > 288.
> > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> > 288.
> > Can't call method "mtime" on an undefined value at /var/ipfire/ids-
> > functions.pl line 1512
> > 
> > Reloading the IPFire browser page and going back to the IDS main
> > page
> > gives the same result with the additional two lines in the log
> > 
> > Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
> > 288.
> > Can't call method "mtime" on an undefined value at /var/ipfire/ids-
> > functions.pl line 1512.
> > 
> 
> Sorry I'm unable to reproduce this - maybe a download error?

Fixed by 
https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=7cf0ecadc14c2a8f6a711ff3ff3dfa2c0a516fb5
and
https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=e59b8178e0cb4098904a8c0f591639d92a1f625e

> 
> > Sorry for breaking it again. If any of my steps are not clear let
> > me
> > know and I will clarify where necessary.
> 
> Hey, this is why we do testing - each found bug until release is a
> good
> bug!
> 
> > 
> > Regards,
> > 
> > Adolf.
> > 
> > 
> > On 11/04/2021 11:49, Adolf Belka wrote:
> > > Hi Stefan,
> > > 
> > > I have installed the new version from scratch in my ipfire vm
> > > testbed. I followed "all" the instructions this time :-)
> > > 
> > > I was able to add additional providers and then go and select the
> > > rules I wanted and had no problems at all.
> > > 
> > > Looks like all fixed. I will do further evaluation of it over the
> > > next few days and let you know how things go for me.
> > > 
> > > Regards,
> > > 
> > > Adolf.
> > > 
> > > On 11/04/2021 10:46, Stefan Schantl wrote:
> > > > Hello again,
> > > > 
> > > > I've tested and uploaded the fourth test verstion.
> > > > 
> > > > https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-004.tar.gz
> > > > 
> > > > This time the ownership of all files are correct at my test
> > > > system.
> > > > 
> > > > (Tested with ruleset changes and without)
> > > > 
> > > > Best regards,
> > > > 
> > > > -Stefan
> > > > 
> > > > > Best regards,
> > > > > 
> > > > > -Stefan
> > > > > 
> > > > > > Hi Stefan,
> > > > > > 
> > > > > > I copied the new tarfile to my ipfire vm testbed machine
> > > > > > and
> > > > > > extracted it and ran the converter script. No errors. I
> > > > > > then
> > > > > > used
> > > > > > the
> > > > > > wui page to add a new provider to the list then selected to
> > > > > > customize
> > > > > > the rules and ticked the box for the added rules. Then I
> > > > > > pressed
> > > > > > apply and got a blank white screen again.
> > > > > > 
> > > > > > 
> > > > > > The error log has the following:-
> > > > > > 
> > > > > > Smartmatch is experimental at /srv/web/ipfire/cgi-
> > > > > > bin/ids.cgi 
> > > > > > line
> > > > > > 288.
> > > > > > Smartmatch is experimental at /srv/web/ipfire/cgi-
> > > > > > bin/ids.cgi 
> > > > > > line
> > > > > > 288.
> > > > > > Smartmatch is experimental at /srv/web/ipfire/cgi-
> > > > > > bin/ids.cgi 
> > > > > > line
> > > > > > 288.
> > > > > > Smartmatch is experimental at /srv/web/ipfire/cgi-
> > > > > > bin/ids.cgi 
> > > > > > line
> > > > > > 288.
> > > > > > Smartmatch is experimental at /srv/web/ipfire/cgi-
> > > > > > bin/ids.cgi 
> > > > > > line
> > > > > > 288.
> > > > > > Smartmatch is experimental at /srv/web/ipfire/cgi-
> > > > > > bin/ids.cgi 
> > > > > > line
> > > > > > 288.
> > > > > > Could not open /var/ipfire/suricata/oinkmaster-provider-
> > > > > > includes.conf. Permission denied
> > > > > > 
> > > > > > 
> > > > > > ls- hal of /var/ipfire/suricata shows the following
> > > > > > 
> > > > > > drwxr-xr-x  2 nobody nobody 4.0K Apr 10 22:47 .
> > > > > > drwxr-xr-x 49 root   root   4.0K Apr  5 08:20 ..
> > > > > > -rw-r--r--  1 nobody nobody    0 Dec 14 19:05 ignored
> > > > > > -rw-r--r--  1 root   root    21K Apr  1 20:00
> > > > > > oinkmaster.conf
> > > > > > -rw-r--r--  1 nobody nobody   61 Apr 10 14:40 oinkmaster-
> > > > > > modify-
> > > > > > sids.conf
> > > > > > -rw-r--r--  1 root   root      0 Apr 10 14:54 oinkmaster-
> > > > > > provider-
> > > > > > includes.conf
> > > > > > -rw-r--r--  1 nobody nobody   55 Apr 10 22:47 providers-
> > > > > > settings
> > > > > > -rw-r--r--  1 root   root   6.0K Apr  5 07:13 ruleset-
> > > > > > sources
> > > > > > -rw-r--r--  1 nobody nobody  102 Apr 10 14:54 settings
> > > > > > -rw-r--r--  1 nobody nobody  140 Apr 10 22:41 suricata-dns-
> > > > > > servers.yaml
> > > > > > -rw-r--r--  1 nobody nobody  125 Apr 10 14:54 suricata-
> > > > > > emerging-
> > > > > > used-
> > > > > > rulefiles.yaml
> > > > > > -rw-r--r--  1 nobody nobody  159 Apr 10 22:41 suricata-
> > > > > > homenet.yaml
> > > > > > -rw-r--r--  1 nobody nobody   98 Apr 10 14:40 suricata-
> > > > > > http-
> > > > > > ports.yaml
> > > > > > -rw-r--r--  1 nobody nobody   95 Apr 10 14:54 suricata-
> > > > > > static-
> > > > > > included-rulefiles.yaml
> > > > > > -rw-r--r--  1 nobody nobody   76 Apr 10 22:47 suricata-
> > > > > > urlhaus-
> > > > > > used-
> > > > > > rulefiles.yaml
> > > > > > -rw-r--r--  1 nobody nobody  214 Apr 10 14:54 suricata-
> > > > > > used-
> > > > > > providers.yaml
> > > > > > 
> > > > > > Three of the files are owned root:root while all the others
> > > > > > are
> > > > > > nobody:nobody
> > > > > > 
> > > > > > 
> > > > > > The above was with extracting and applying the updated tar
> > > > > > file on
> > > > > > top of IPFire after running the last version.
> > > > > > 
> > > > > > I will do a fresh clone of my IPFire vm and then repeat the
> > > > > > tar
> > > > > > extraction and convert and see if that gives any
> > > > > > difference.
> > > > > > 
> > > > > > 
> > > > > > Regards,
> > > > > > 
> > > > > > Adolf
> > > > > > 
> > > > > > On 10/04/2021 20:25, Stefan Schantl wrote:
> > > > > > > Hello list followers,
> > > > > > > 
> > > > > > > after getting a lot of feedback and bug reports I'm happy
> > > > > > > to
> > > > > > > announce the third test version for the new IDS system.
> > > > > > > 
> > > > > > > https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-003.tar.gz
> > > > > > > 
> > > > > > > If you just join testing, please omit the installation
> > > > > > > instructions
> > > > > > > from the initial Mail from this list.
> > > > > > > 
> > > > > > > The converter script now works as expected and runs very
> > > > > > > smooth.
> > > > > > > 
> > > > > > > As usual please post your feedback and opinions to this
> > > > > > > list and
> > > > > > > any
> > > > > > > remain bugs to our bugtracker. (
> > > > > > > https://bugzilla.ipfire.org
> > > > > > > )
> > > > > > > 
> > > > > > > A big thanks in advance,
> > > > > > > 
> > > > > > > -Stefan
> > > > > > > 

Re: IDS with support for multiple ruleset providers

[Stefan Schantl]

[-- Attachment #1: Type: text/plain, Size: 338 bytes --]

Hello Michael, Hello Adolf, Hello *,

I've uploaded the fifth test version.

https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-005.tar.gz

All reported bugs of the previous versions are fixed now and I've
rebased the language files to Core update 155. (master branch).

Best regards,

-Stefan 

Re: IDS with support for multiple ruleset providers

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 8935 bytes --]



> On 14 Apr 2021, at 20:16, Stefan Schantl <stefan.schantl(a)ipfire.org> wrote:
> 
> Am Dienstag, den 13.04.2021, 20:57 +0200 schrieb Stefan Schantl:
>> Hello Adolf,
>> 
>> thanks you very much for your huge effort in testing this and
>> providing
>> this very detailed feedback.
>> 
>> While reading through your single steps it feels sometimes near to
>> get
>> a knot inside my brain....
>> 
>>> Hi Stefan,
>>> 
>>> I did a bit more testing.
>>> 
>>> I added the snort community rules set. I then went to customise and
>>> left the snort rules unchecked then pressed apply.
>>> 
>>> I then disabled the snort rules from the main page and on the
>>> customise page the snort rules were no longer showing.
>> 
>> Workes as designed.
>> 
>>> I then enabled the snort rules on the first page and then went to
>>> customise but the snort rules still were not showing.
>> 
>> Very good catch - Fixed.
>> 
>>> I deleted the snort ruleset provider on the first page and then
>>> added
>>> them back and now the snort ruleset was shown again on the
>>> customise
>>> page.
>> 
>> OK.
>> 
>>> I then checked the snort ruleset and applied it and then entered
>>> customise again and unchecked the snort ruleset and applied it.
>>> When
>>> I went back into customise the snort ruleset was checked again. So
>>> once checked I could not uncheck it and keep it that why by
>>> pressing
>>> apply.
>>> 
>> 
>> Confirmed. Thanks for finding this.
> 
> Fixed by commit:
> 
> https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=79cc92267f1811beab84ae190fc1c3724a67e5f4
> 
>> 
>>> I then deleted the snort ruleset provider from the first page. Then
>>> the ruleset was gone from the customise page.
>>> 
>>> Then I added the snort ruleset provider back in but then got an
>>> error
>>> message saying that the snort ruleset provider was already
>>> selected.
>>> I then pressed back and came back to the main page with no snort
>>> ruleset provider but also with the page  only showing down to the
>>> Ruleset Settings table. There was nothing else after that.
>>> 
>>> The httpd/error_log showed the following
>>> 
>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>> 288.
>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>> 288.
>>> Can't call method "mtime" on an undefined value at /var/ipfire/ids-
>>> functions.pl line 1512
>>> 
>>> Reloading the IPFire browser page and going back to the IDS main
>>> page
>>> gives the same result with the additional two lines in the log
>>> 
>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>> 288.
>>> Can't call method "mtime" on an undefined value at /var/ipfire/ids-
>>> functions.pl line 1512.
>>> 
>> 
>> Sorry I'm unable to reproduce this - maybe a download error?
> 
> Fixed by 
> https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=7cf0ecadc14c2a8f6a711ff3ff3dfa2c0a516fb5
> and
> https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=e59b8178e0cb4098904a8c0f591639d92a1f625e

I do not think that the second patch fixes the problem. You are still using the return value of stat() here:

   my $mtime = $stat->mtime;

This might be set to “undefined” and localtime() and strftime() might return undefined as well, but you could simply return “N/A” just after calling stat.

> 
>> 
>>> Sorry for breaking it again. If any of my steps are not clear let
>>> me
>>> know and I will clarify where necessary.
>> 
>> Hey, this is why we do testing - each found bug until release is a
>> good
>> bug!
>> 
>>> 
>>> Regards,
>>> 
>>> Adolf.
>>> 
>>> 
>>> On 11/04/2021 11:49, Adolf Belka wrote:
>>>> Hi Stefan,
>>>> 
>>>> I have installed the new version from scratch in my ipfire vm
>>>> testbed. I followed "all" the instructions this time :-)
>>>> 
>>>> I was able to add additional providers and then go and select the
>>>> rules I wanted and had no problems at all.
>>>> 
>>>> Looks like all fixed. I will do further evaluation of it over the
>>>> next few days and let you know how things go for me.
>>>> 
>>>> Regards,
>>>> 
>>>> Adolf.
>>>> 
>>>> On 11/04/2021 10:46, Stefan Schantl wrote:
>>>>> Hello again,
>>>>> 
>>>>> I've tested and uploaded the fourth test verstion.
>>>>> 
>>>>> https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-004.tar.gz
>>>>> 
>>>>> This time the ownership of all files are correct at my test
>>>>> system.
>>>>> 
>>>>> (Tested with ruleset changes and without)
>>>>> 
>>>>> Best regards,
>>>>> 
>>>>> -Stefan
>>>>> 
>>>>>> Best regards,
>>>>>> 
>>>>>> -Stefan
>>>>>> 
>>>>>>> Hi Stefan,
>>>>>>> 
>>>>>>> I copied the new tarfile to my ipfire vm testbed machine
>>>>>>> and
>>>>>>> extracted it and ran the converter script. No errors. I
>>>>>>> then
>>>>>>> used
>>>>>>> the
>>>>>>> wui page to add a new provider to the list then selected to
>>>>>>> customize
>>>>>>> the rules and ticked the box for the added rules. Then I
>>>>>>> pressed
>>>>>>> apply and got a blank white screen again.
>>>>>>> 
>>>>>>> 
>>>>>>> The error log has the following:-
>>>>>>> 
>>>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-
>>>>>>> bin/ids.cgi 
>>>>>>> line
>>>>>>> 288.
>>>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-
>>>>>>> bin/ids.cgi 
>>>>>>> line
>>>>>>> 288.
>>>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-
>>>>>>> bin/ids.cgi 
>>>>>>> line
>>>>>>> 288.
>>>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-
>>>>>>> bin/ids.cgi 
>>>>>>> line
>>>>>>> 288.
>>>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-
>>>>>>> bin/ids.cgi 
>>>>>>> line
>>>>>>> 288.
>>>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-
>>>>>>> bin/ids.cgi 
>>>>>>> line
>>>>>>> 288.
>>>>>>> Could not open /var/ipfire/suricata/oinkmaster-provider-
>>>>>>> includes.conf. Permission denied
>>>>>>> 
>>>>>>> 
>>>>>>> ls- hal of /var/ipfire/suricata shows the following
>>>>>>> 
>>>>>>> drwxr-xr-x  2 nobody nobody 4.0K Apr 10 22:47 .
>>>>>>> drwxr-xr-x 49 root   root   4.0K Apr  5 08:20 ..
>>>>>>> -rw-r--r--  1 nobody nobody    0 Dec 14 19:05 ignored
>>>>>>> -rw-r--r--  1 root   root    21K Apr  1 20:00
>>>>>>> oinkmaster.conf
>>>>>>> -rw-r--r--  1 nobody nobody   61 Apr 10 14:40 oinkmaster-
>>>>>>> modify-
>>>>>>> sids.conf
>>>>>>> -rw-r--r--  1 root   root      0 Apr 10 14:54 oinkmaster-
>>>>>>> provider-
>>>>>>> includes.conf
>>>>>>> -rw-r--r--  1 nobody nobody   55 Apr 10 22:47 providers-
>>>>>>> settings
>>>>>>> -rw-r--r--  1 root   root   6.0K Apr  5 07:13 ruleset-
>>>>>>> sources
>>>>>>> -rw-r--r--  1 nobody nobody  102 Apr 10 14:54 settings
>>>>>>> -rw-r--r--  1 nobody nobody  140 Apr 10 22:41 suricata-dns-
>>>>>>> servers.yaml
>>>>>>> -rw-r--r--  1 nobody nobody  125 Apr 10 14:54 suricata-
>>>>>>> emerging-
>>>>>>> used-
>>>>>>> rulefiles.yaml
>>>>>>> -rw-r--r--  1 nobody nobody  159 Apr 10 22:41 suricata-
>>>>>>> homenet.yaml
>>>>>>> -rw-r--r--  1 nobody nobody   98 Apr 10 14:40 suricata-
>>>>>>> http-
>>>>>>> ports.yaml
>>>>>>> -rw-r--r--  1 nobody nobody   95 Apr 10 14:54 suricata-
>>>>>>> static-
>>>>>>> included-rulefiles.yaml
>>>>>>> -rw-r--r--  1 nobody nobody   76 Apr 10 22:47 suricata-
>>>>>>> urlhaus-
>>>>>>> used-
>>>>>>> rulefiles.yaml
>>>>>>> -rw-r--r--  1 nobody nobody  214 Apr 10 14:54 suricata-
>>>>>>> used-
>>>>>>> providers.yaml
>>>>>>> 
>>>>>>> Three of the files are owned root:root while all the others
>>>>>>> are
>>>>>>> nobody:nobody
>>>>>>> 
>>>>>>> 
>>>>>>> The above was with extracting and applying the updated tar
>>>>>>> file on
>>>>>>> top of IPFire after running the last version.
>>>>>>> 
>>>>>>> I will do a fresh clone of my IPFire vm and then repeat the
>>>>>>> tar
>>>>>>> extraction and convert and see if that gives any
>>>>>>> difference.
>>>>>>> 
>>>>>>> 
>>>>>>> Regards,
>>>>>>> 
>>>>>>> Adolf
>>>>>>> 
>>>>>>> On 10/04/2021 20:25, Stefan Schantl wrote:
>>>>>>>> Hello list followers,
>>>>>>>> 
>>>>>>>> after getting a lot of feedback and bug reports I'm happy
>>>>>>>> to
>>>>>>>> announce the third test version for the new IDS system.
>>>>>>>> 
>>>>>>>> https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-003.tar.gz
>>>>>>>> 
>>>>>>>> If you just join testing, please omit the installation
>>>>>>>> instructions
>>>>>>>> from the initial Mail from this list.
>>>>>>>> 
>>>>>>>> The converter script now works as expected and runs very
>>>>>>>> smooth.
>>>>>>>> 
>>>>>>>> As usual please post your feedback and opinions to this
>>>>>>>> list and
>>>>>>>> any
>>>>>>>> remain bugs to our bugtracker. (
>>>>>>>> https://bugzilla.ipfire.org
>>>>>>>> )
>>>>>>>> 
>>>>>>>> A big thanks in advance,
>>>>>>>> 
>>>>>>>> -Stefan