Re: IDS with support for multiple ruleset providers

[Michael Tremer <michael.tremer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 8935 bytes --]



> On 14 Apr 2021, at 20:16, Stefan Schantl <stefan.schantl(a)ipfire.org> wrote:
> 
> Am Dienstag, den 13.04.2021, 20:57 +0200 schrieb Stefan Schantl:
>> Hello Adolf,
>> 
>> thanks you very much for your huge effort in testing this and
>> providing
>> this very detailed feedback.
>> 
>> While reading through your single steps it feels sometimes near to
>> get
>> a knot inside my brain....
>> 
>>> Hi Stefan,
>>> 
>>> I did a bit more testing.
>>> 
>>> I added the snort community rules set. I then went to customise and
>>> left the snort rules unchecked then pressed apply.
>>> 
>>> I then disabled the snort rules from the main page and on the
>>> customise page the snort rules were no longer showing.
>> 
>> Workes as designed.
>> 
>>> I then enabled the snort rules on the first page and then went to
>>> customise but the snort rules still were not showing.
>> 
>> Very good catch - Fixed.
>> 
>>> I deleted the snort ruleset provider on the first page and then
>>> added
>>> them back and now the snort ruleset was shown again on the
>>> customise
>>> page.
>> 
>> OK.
>> 
>>> I then checked the snort ruleset and applied it and then entered
>>> customise again and unchecked the snort ruleset and applied it.
>>> When
>>> I went back into customise the snort ruleset was checked again. So
>>> once checked I could not uncheck it and keep it that why by
>>> pressing
>>> apply.
>>> 
>> 
>> Confirmed. Thanks for finding this.
> 
> Fixed by commit:
> 
> https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=79cc92267f1811beab84ae190fc1c3724a67e5f4
> 
>> 
>>> I then deleted the snort ruleset provider from the first page. Then
>>> the ruleset was gone from the customise page.
>>> 
>>> Then I added the snort ruleset provider back in but then got an
>>> error
>>> message saying that the snort ruleset provider was already
>>> selected.
>>> I then pressed back and came back to the main page with no snort
>>> ruleset provider but also with the page  only showing down to the
>>> Ruleset Settings table. There was nothing else after that.
>>> 
>>> The httpd/error_log showed the following
>>> 
>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>> 288.
>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>> 288.
>>> Can't call method "mtime" on an undefined value at /var/ipfire/ids-
>>> functions.pl line 1512
>>> 
>>> Reloading the IPFire browser page and going back to the IDS main
>>> page
>>> gives the same result with the additional two lines in the log
>>> 
>>> Smartmatch is experimental at /srv/web/ipfire/cgi-bin/ids.cgi line
>>> 288.
>>> Can't call method "mtime" on an undefined value at /var/ipfire/ids-
>>> functions.pl line 1512.
>>> 
>> 
>> Sorry I'm unable to reproduce this - maybe a download error?
> 
> Fixed by 
> https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=7cf0ecadc14c2a8f6a711ff3ff3dfa2c0a516fb5
> and
> https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=e59b8178e0cb4098904a8c0f591639d92a1f625e

I do not think that the second patch fixes the problem. You are still using the return value of stat() here:

   my $mtime = $stat->mtime;

This might be set to “undefined” and localtime() and strftime() might return undefined as well, but you could simply return “N/A” just after calling stat.

> 
>> 
>>> Sorry for breaking it again. If any of my steps are not clear let
>>> me
>>> know and I will clarify where necessary.
>> 
>> Hey, this is why we do testing - each found bug until release is a
>> good
>> bug!
>> 
>>> 
>>> Regards,
>>> 
>>> Adolf.
>>> 
>>> 
>>> On 11/04/2021 11:49, Adolf Belka wrote:
>>>> Hi Stefan,
>>>> 
>>>> I have installed the new version from scratch in my ipfire vm
>>>> testbed. I followed "all" the instructions this time :-)
>>>> 
>>>> I was able to add additional providers and then go and select the
>>>> rules I wanted and had no problems at all.
>>>> 
>>>> Looks like all fixed. I will do further evaluation of it over the
>>>> next few days and let you know how things go for me.
>>>> 
>>>> Regards,
>>>> 
>>>> Adolf.
>>>> 
>>>> On 11/04/2021 10:46, Stefan Schantl wrote:
>>>>> Hello again,
>>>>> 
>>>>> I've tested and uploaded the fourth test verstion.
>>>>> 
>>>>> https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-004.tar.gz
>>>>> 
>>>>> This time the ownership of all files are correct at my test
>>>>> system.
>>>>> 
>>>>> (Tested with ruleset changes and without)
>>>>> 
>>>>> Best regards,
>>>>> 
>>>>> -Stefan
>>>>> 
>>>>>> Best regards,
>>>>>> 
>>>>>> -Stefan
>>>>>> 
>>>>>>> Hi Stefan,
>>>>>>> 
>>>>>>> I copied the new tarfile to my ipfire vm testbed machine
>>>>>>> and
>>>>>>> extracted it and ran the converter script. No errors. I
>>>>>>> then
>>>>>>> used
>>>>>>> the
>>>>>>> wui page to add a new provider to the list then selected to
>>>>>>> customize
>>>>>>> the rules and ticked the box for the added rules. Then I
>>>>>>> pressed
>>>>>>> apply and got a blank white screen again.
>>>>>>> 
>>>>>>> 
>>>>>>> The error log has the following:-
>>>>>>> 
>>>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-
>>>>>>> bin/ids.cgi 
>>>>>>> line
>>>>>>> 288.
>>>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-
>>>>>>> bin/ids.cgi 
>>>>>>> line
>>>>>>> 288.
>>>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-
>>>>>>> bin/ids.cgi 
>>>>>>> line
>>>>>>> 288.
>>>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-
>>>>>>> bin/ids.cgi 
>>>>>>> line
>>>>>>> 288.
>>>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-
>>>>>>> bin/ids.cgi 
>>>>>>> line
>>>>>>> 288.
>>>>>>> Smartmatch is experimental at /srv/web/ipfire/cgi-
>>>>>>> bin/ids.cgi 
>>>>>>> line
>>>>>>> 288.
>>>>>>> Could not open /var/ipfire/suricata/oinkmaster-provider-
>>>>>>> includes.conf. Permission denied
>>>>>>> 
>>>>>>> 
>>>>>>> ls- hal of /var/ipfire/suricata shows the following
>>>>>>> 
>>>>>>> drwxr-xr-x  2 nobody nobody 4.0K Apr 10 22:47 .
>>>>>>> drwxr-xr-x 49 root   root   4.0K Apr  5 08:20 ..
>>>>>>> -rw-r--r--  1 nobody nobody    0 Dec 14 19:05 ignored
>>>>>>> -rw-r--r--  1 root   root    21K Apr  1 20:00
>>>>>>> oinkmaster.conf
>>>>>>> -rw-r--r--  1 nobody nobody   61 Apr 10 14:40 oinkmaster-
>>>>>>> modify-
>>>>>>> sids.conf
>>>>>>> -rw-r--r--  1 root   root      0 Apr 10 14:54 oinkmaster-
>>>>>>> provider-
>>>>>>> includes.conf
>>>>>>> -rw-r--r--  1 nobody nobody   55 Apr 10 22:47 providers-
>>>>>>> settings
>>>>>>> -rw-r--r--  1 root   root   6.0K Apr  5 07:13 ruleset-
>>>>>>> sources
>>>>>>> -rw-r--r--  1 nobody nobody  102 Apr 10 14:54 settings
>>>>>>> -rw-r--r--  1 nobody nobody  140 Apr 10 22:41 suricata-dns-
>>>>>>> servers.yaml
>>>>>>> -rw-r--r--  1 nobody nobody  125 Apr 10 14:54 suricata-
>>>>>>> emerging-
>>>>>>> used-
>>>>>>> rulefiles.yaml
>>>>>>> -rw-r--r--  1 nobody nobody  159 Apr 10 22:41 suricata-
>>>>>>> homenet.yaml
>>>>>>> -rw-r--r--  1 nobody nobody   98 Apr 10 14:40 suricata-
>>>>>>> http-
>>>>>>> ports.yaml
>>>>>>> -rw-r--r--  1 nobody nobody   95 Apr 10 14:54 suricata-
>>>>>>> static-
>>>>>>> included-rulefiles.yaml
>>>>>>> -rw-r--r--  1 nobody nobody   76 Apr 10 22:47 suricata-
>>>>>>> urlhaus-
>>>>>>> used-
>>>>>>> rulefiles.yaml
>>>>>>> -rw-r--r--  1 nobody nobody  214 Apr 10 14:54 suricata-
>>>>>>> used-
>>>>>>> providers.yaml
>>>>>>> 
>>>>>>> Three of the files are owned root:root while all the others
>>>>>>> are
>>>>>>> nobody:nobody
>>>>>>> 
>>>>>>> 
>>>>>>> The above was with extracting and applying the updated tar
>>>>>>> file on
>>>>>>> top of IPFire after running the last version.
>>>>>>> 
>>>>>>> I will do a fresh clone of my IPFire vm and then repeat the
>>>>>>> tar
>>>>>>> extraction and convert and see if that gives any
>>>>>>> difference.
>>>>>>> 
>>>>>>> 
>>>>>>> Regards,
>>>>>>> 
>>>>>>> Adolf
>>>>>>> 
>>>>>>> On 10/04/2021 20:25, Stefan Schantl wrote:
>>>>>>>> Hello list followers,
>>>>>>>> 
>>>>>>>> after getting a lot of feedback and bug reports I'm happy
>>>>>>>> to
>>>>>>>> announce the third test version for the new IDS system.
>>>>>>>> 
>>>>>>>> https://people.ipfire.org/~stevee/ids-multiple-providers/ids-multiple-providers-003.tar.gz
>>>>>>>> 
>>>>>>>> If you just join testing, please omit the installation
>>>>>>>> instructions
>>>>>>>> from the initial Mail from this list.
>>>>>>>> 
>>>>>>>> The converter script now works as expected and runs very
>>>>>>>> smooth.
>>>>>>>> 
>>>>>>>> As usual please post your feedback and opinions to this
>>>>>>>> list and
>>>>>>>> any
>>>>>>>> remain bugs to our bugtracker. (
>>>>>>>> https://bugzilla.ipfire.org
>>>>>>>> )
>>>>>>>> 
>>>>>>>> A big thanks in advance,
>>>>>>>> 
>>>>>>>> -Stefan