Re: New IDS won't unlink "/tmp/ids_page_locked"

Re: New IDS won't unlink "/tmp/ids_page_locked"

[Stefan Schantl]

[-- Attachment #1: Type: text/plain, Size: 3965 bytes --]

Hello Mathias,

thanks for reporting your issue here.

The IDS page locking issue should be fixed and shipped with the final
C164 release.

See:
https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=85b1d83b2a6fe2beb8169f3e810e915c4ad54036

Please report if you are affected again by this issue.

You safely can drop those folder and files from "/tmp".

Best regards,

-Stefan
> Hi,
> 
> ..I'd like to have *two* problems with the IDS/IPS... ;-)
> 
> 
> 1. After upgrading to Core164 yesterday I took a look at the IDS GUI
> today and found it locked with a spinning wheel at "Ruleset update in
> progress. Please wait until all operations have completed
> successfully..."
> 
> Despite this message IPS seemed to work.
> 
> So I searched and found that the file "/tmp/ids_page_locked" still
> existed. After deleting this file manually the web GUI opened,
> everything seems normal.
> 
> Three activated rulesets were running, see attachment "01".
> 
> But the Web GUI stopped because of lines 234ff in 'ids.cgi':
> 
> ...
> # Check if the page is locked, in this case, the ids_page_lock_file
> exists.
> if (-e $IDS::ids_page_lock_file) {
>         # Lock the webpage and print notice about autoupgrade of the
> ruleset
>         # is in progess.
>         &working_notice("$Lang::tr{'ids ruleset autoupdate in
> progress'}");
> 
>         # Loop and check if the file still exists.
>         while(-e $IDS::ids_page_lock_file) {
>                 # Sleep for a second and re-check.
>                 sleep 1;
>         }
> ...
> 
> I searched again and found that there are two subroutines in
> '/var/ipfire/ids-functions.pl', lines 1920ff, which are responsible
> for
> locking and unlocking this file:
> 
> ...
> ## Function to write the lock file for locking the WUI, while
> ## the autoupdate script runs.
> #
> sub lock_ids_page() {
>         # Call subfunction to create the file.
>         &create_empty_file($ids_page_lock_file);
> }
> 
> #
> ## Function to release the lock of the WUI, again.
> #
> sub unlock_ids_page() {
>         # Delete lock file.
>         unlink($ids_page_lock_file);
> }
> ...
> 
> Somehow lines 103ff in "/usr/local/bin/update-ids-ruleset" didn't
> work/weren't executed(?):
> 
> ...
> # Lock the IDS page.
> &IDS::lock_ids_page();
> 
> # The script has requested a lock, so set locket to "1".
> $locked = "1";
> 
> # Grab the configured providers.
> &General::readhasharray("$IDS::providers_settings_file",
> \%providers);
> 
> # Loop through the array of available providers.
> foreach my $id (keys %providers) {
>    # Assign some nice variabled.
>    my $provider = $providers{$id}[0];
>    my $autoupdate_status = $providers{$id}[3];
> 
>    # Skip the provider if autoupdate is not enabled.
>    next unless($autoupdate_status eq "enabled");
> 
>    # Call the download function and gather the new ruleset for the
> current processed provider.
>    if(&IDS::downloadruleset($provider)) {
>       # Store error message for displaying in the WUI.
>       &IDS::_store_error_message("$provider: $Lang::tr{'could not
> download latest updates'}");
> 
>       # Unlock the IDS page.
>       &IDS::unlock_ids_page();
> 
>       # Exit.
>       exit 0;
>    }
> ...
> 
> Manual updates - yesterday - were ok, but the automatic updates
> tonight
> didn't unlink the lock file: IDS GUI stopped at the next call today.
> 
> ##########
> 
> 2. There still exist a directory '/tmp/isd_tmp' containing 'conf'
> ('config' and 'map' files from emerging-threats) and a 'rules'-dir
> containing all '*.rules'-files. Leftovers!? Can I safely delete these
> dirs?
> 
> And: can anyone confirm these two problems - or reproduce?
> 
> Best,
> Matthias

Re: New IDS won't unlink "/tmp/ids_page_locked"

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 4552 bytes --]

On 14.03.2022 06:08, Stefan Schantl wrote:
> Hello Mathias,

Hi Stefan,

> thanks for reporting your issue here.

No problem - I should have taken a closer look: 'update-ids-ruleset'
seems to be missing in the Core update for Core164, I was still using
the old version from Core163. ;-)

See:
https://git.ipfire.org/?p=ipfire-2.x.git;a=tree;f=config/rootfiles/core/164/filelists;h=c81a06294cd3527b9b68d1352d487bf1c8f95c9e;hb=refs/heads/core164

> The IDS page locking issue should be fixed and shipped with the final
> C164 release.
> 
> See:
> https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=85b1d83b2a6fe2beb8169f3e810e915c4ad54036
> 
> Please report if you are affected again by this issue.

I'll test and report.

Best,
Matthias

> You safely can drop those folder and files from "/tmp".
> 
> Best regards,
> 
> -Stefan
>> Hi,
>> 
>> ..I'd like to have *two* problems with the IDS/IPS... ;-)
>> 
>> 
>> 1. After upgrading to Core164 yesterday I took a look at the IDS GUI
>> today and found it locked with a spinning wheel at "Ruleset update in
>> progress. Please wait until all operations have completed
>> successfully..."
>> 
>> Despite this message IPS seemed to work.
>> 
>> So I searched and found that the file "/tmp/ids_page_locked" still
>> existed. After deleting this file manually the web GUI opened,
>> everything seems normal.
>> 
>> Three activated rulesets were running, see attachment "01".
>> 
>> But the Web GUI stopped because of lines 234ff in 'ids.cgi':
>> 
>> ...
>> # Check if the page is locked, in this case, the ids_page_lock_file
>> exists.
>> if (-e $IDS::ids_page_lock_file) {
>>         # Lock the webpage and print notice about autoupgrade of the
>> ruleset
>>         # is in progess.
>>         &working_notice("$Lang::tr{'ids ruleset autoupdate in
>> progress'}");
>> 
>>         # Loop and check if the file still exists.
>>         while(-e $IDS::ids_page_lock_file) {
>>                 # Sleep for a second and re-check.
>>                 sleep 1;
>>         }
>> ...
>> 
>> I searched again and found that there are two subroutines in
>> '/var/ipfire/ids-functions.pl', lines 1920ff, which are responsible
>> for
>> locking and unlocking this file:
>> 
>> ...
>> ## Function to write the lock file for locking the WUI, while
>> ## the autoupdate script runs.
>> #
>> sub lock_ids_page() {
>>         # Call subfunction to create the file.
>>         &create_empty_file($ids_page_lock_file);
>> }
>> 
>> #
>> ## Function to release the lock of the WUI, again.
>> #
>> sub unlock_ids_page() {
>>         # Delete lock file.
>>         unlink($ids_page_lock_file);
>> }
>> ...
>> 
>> Somehow lines 103ff in "/usr/local/bin/update-ids-ruleset" didn't
>> work/weren't executed(?):
>> 
>> ...
>> # Lock the IDS page.
>> &IDS::lock_ids_page();
>> 
>> # The script has requested a lock, so set locket to "1".
>> $locked = "1";
>> 
>> # Grab the configured providers.
>> &General::readhasharray("$IDS::providers_settings_file",
>> \%providers);
>> 
>> # Loop through the array of available providers.
>> foreach my $id (keys %providers) {
>>    # Assign some nice variabled.
>>    my $provider = $providers{$id}[0];
>>    my $autoupdate_status = $providers{$id}[3];
>> 
>>    # Skip the provider if autoupdate is not enabled.
>>    next unless($autoupdate_status eq "enabled");
>> 
>>    # Call the download function and gather the new ruleset for the
>> current processed provider.
>>    if(&IDS::downloadruleset($provider)) {
>>       # Store error message for displaying in the WUI.
>>       &IDS::_store_error_message("$provider: $Lang::tr{'could not
>> download latest updates'}");
>> 
>>       # Unlock the IDS page.
>>       &IDS::unlock_ids_page();
>> 
>>       # Exit.
>>       exit 0;
>>    }
>> ...
>> 
>> Manual updates - yesterday - were ok, but the automatic updates
>> tonight
>> didn't unlink the lock file: IDS GUI stopped at the next call today.
>> 
>> ##########
>> 
>> 2. There still exist a directory '/tmp/isd_tmp' containing 'conf'
>> ('config' and 'map' files from emerging-threats) and a 'rules'-dir
>> containing all '*.rules'-files. Leftovers!? Can I safely delete these
>> dirs?
>> 
>> And: can anyone confirm these two problems - or reproduce?
>> 
>> Best,
>> Matthias
> 
> 

Re: New IDS won't unlink "/tmp/ids_page_locked"

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 5878 bytes --]

Hi,

after updating the old '/usr/local/bin/update-ids-ruleset' to one from
Core164, I manually started "Force ruleset update" through IDS GUI for
one of my lists.

Result:
GUI changes to "Downloading and unpacking new ruleset. Please wait until
all operations have completed successfully...". Then it stays there
forever until I reload it - which works after some time.

But in '/var/tmp' there are remains of a 'ids_tmp'-directory, containing
a 'conf' and a 'rules' directory. These seem to contain the updated,
unpacked config- and rule-files(?). For me it looks as if in
'/var/lib/suricata' there are still the *old* rule files.

'var/lib/suricata' and '/tmp/ids_tmp/rules' contain identically named
*.rules-files, but the rules-files in '/var/lib/suricata' start with
"drop tcp ..." and the ones in '/var/ids_tmp/rules' with "alert tcp ...".

And 'suricata' isn't restarted, either. No update entries in the logs.

Besides that, the GUI shows the updated date and time stamps from the
downloaded archives in '/var/tmp'. At first look, everything seems ok,
but it isn't. Weird...

On 14.03.2022 18:04, Matthias Fischer wrote:
> On 14.03.2022 06:08, Stefan Schantl wrote:
>> Hello Mathias,
> 
> Hi Stefan,
> 
>> thanks for reporting your issue here.
> 
> No problem - I should have taken a closer look: 'update-ids-ruleset'
> seems to be missing in the Core update for Core164, I was still using
> the old version from Core163. ;-)
> 
> See:
> https://git.ipfire.org/?p=ipfire-2.x.git;a=tree;f=config/rootfiles/core/164/filelists;h=c81a06294cd3527b9b68d1352d487bf1c8f95c9e;hb=refs/heads/core164
> 
>> The IDS page locking issue should be fixed and shipped with the final
>> C164 release.
>> 
>> See:
>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=85b1d83b2a6fe2beb8169f3e810e915c4ad54036
>> 
>> Please report if you are affected again by this issue.
> 
> I'll test and report.
> 
> Best,
> Matthias
> 
>> You safely can drop those folder and files from "/tmp".
>> 
>> Best regards,
>> 
>> -Stefan
>>> Hi,
>>> 
>>> ..I'd like to have *two* problems with the IDS/IPS... ;-)
>>> 
>>> 
>>> 1. After upgrading to Core164 yesterday I took a look at the IDS GUI
>>> today and found it locked with a spinning wheel at "Ruleset update in
>>> progress. Please wait until all operations have completed
>>> successfully..."
>>> 
>>> Despite this message IPS seemed to work.
>>> 
>>> So I searched and found that the file "/tmp/ids_page_locked" still
>>> existed. After deleting this file manually the web GUI opened,
>>> everything seems normal.
>>> 
>>> Three activated rulesets were running, see attachment "01".
>>> 
>>> But the Web GUI stopped because of lines 234ff in 'ids.cgi':
>>> 
>>> ...
>>> # Check if the page is locked, in this case, the ids_page_lock_file
>>> exists.
>>> if (-e $IDS::ids_page_lock_file) {
>>>         # Lock the webpage and print notice about autoupgrade of the
>>> ruleset
>>>         # is in progess.
>>>         &working_notice("$Lang::tr{'ids ruleset autoupdate in
>>> progress'}");
>>> 
>>>         # Loop and check if the file still exists.
>>>         while(-e $IDS::ids_page_lock_file) {
>>>                 # Sleep for a second and re-check.
>>>                 sleep 1;
>>>         }
>>> ...
>>> 
>>> I searched again and found that there are two subroutines in
>>> '/var/ipfire/ids-functions.pl', lines 1920ff, which are responsible
>>> for
>>> locking and unlocking this file:
>>> 
>>> ...
>>> ## Function to write the lock file for locking the WUI, while
>>> ## the autoupdate script runs.
>>> #
>>> sub lock_ids_page() {
>>>         # Call subfunction to create the file.
>>>         &create_empty_file($ids_page_lock_file);
>>> }
>>> 
>>> #
>>> ## Function to release the lock of the WUI, again.
>>> #
>>> sub unlock_ids_page() {
>>>         # Delete lock file.
>>>         unlink($ids_page_lock_file);
>>> }
>>> ...
>>> 
>>> Somehow lines 103ff in "/usr/local/bin/update-ids-ruleset" didn't
>>> work/weren't executed(?):
>>> 
>>> ...
>>> # Lock the IDS page.
>>> &IDS::lock_ids_page();
>>> 
>>> # The script has requested a lock, so set locket to "1".
>>> $locked = "1";
>>> 
>>> # Grab the configured providers.
>>> &General::readhasharray("$IDS::providers_settings_file",
>>> \%providers);
>>> 
>>> # Loop through the array of available providers.
>>> foreach my $id (keys %providers) {
>>>    # Assign some nice variabled.
>>>    my $provider = $providers{$id}[0];
>>>    my $autoupdate_status = $providers{$id}[3];
>>> 
>>>    # Skip the provider if autoupdate is not enabled.
>>>    next unless($autoupdate_status eq "enabled");
>>> 
>>>    # Call the download function and gather the new ruleset for the
>>> current processed provider.
>>>    if(&IDS::downloadruleset($provider)) {
>>>       # Store error message for displaying in the WUI.
>>>       &IDS::_store_error_message("$provider: $Lang::tr{'could not
>>> download latest updates'}");
>>> 
>>>       # Unlock the IDS page.
>>>       &IDS::unlock_ids_page();
>>> 
>>>       # Exit.
>>>       exit 0;
>>>    }
>>> ...
>>> 
>>> Manual updates - yesterday - were ok, but the automatic updates
>>> tonight
>>> didn't unlink the lock file: IDS GUI stopped at the next call today.
>>> 
>>> ##########
>>> 
>>> 2. There still exist a directory '/tmp/isd_tmp' containing 'conf'
>>> ('config' and 'map' files from emerging-threats) and a 'rules'-dir
>>> containing all '*.rules'-files. Leftovers!? Can I safely delete these
>>> dirs?
>>> 
>>> And: can anyone confirm these two problems - or reproduce?
>>> 
>>> Best,
>>> Matthias
>> 
>> 
> 

Re: New IDS won't unlink "/tmp/ids_page_locked"

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 6448 bytes --]

Hi,

I wanted to report that after updating 'update-ids-ruleset' the problems
are gone:

- GUI doesn't hang anymore.
- After the daily rules update, '/tmp' doesn't contain any leftovers.
- Reloading the rules did work and was logged properly.

Just in case someone wanted to know... ;-)

Best,
Matthias

On 14.03.2022 19:01, Matthias Fischer wrote:
> Hi,
> 
> after updating the old '/usr/local/bin/update-ids-ruleset' to one from
> Core164, I manually started "Force ruleset update" through IDS GUI for
> one of my lists.
> 
> Result:
> GUI changes to "Downloading and unpacking new ruleset. Please wait until
> all operations have completed successfully...". Then it stays there
> forever until I reload it - which works after some time.
> 
> But in '/var/tmp' there are remains of a 'ids_tmp'-directory, containing
> a 'conf' and a 'rules' directory. These seem to contain the updated,
> unpacked config- and rule-files(?). For me it looks as if in
> '/var/lib/suricata' there are still the *old* rule files.
> 
> 'var/lib/suricata' and '/tmp/ids_tmp/rules' contain identically named
> *.rules-files, but the rules-files in '/var/lib/suricata' start with
> "drop tcp ..." and the ones in '/var/ids_tmp/rules' with "alert tcp ...".
> 
> And 'suricata' isn't restarted, either. No update entries in the logs.
> 
> Besides that, the GUI shows the updated date and time stamps from the
> downloaded archives in '/var/tmp'. At first look, everything seems ok,
> but it isn't. Weird...
> 
> On 14.03.2022 18:04, Matthias Fischer wrote:
>> On 14.03.2022 06:08, Stefan Schantl wrote:
>>> Hello Mathias,
>> 
>> Hi Stefan,
>> 
>>> thanks for reporting your issue here.
>> 
>> No problem - I should have taken a closer look: 'update-ids-ruleset'
>> seems to be missing in the Core update for Core164, I was still using
>> the old version from Core163. ;-)
>> 
>> See:
>> https://git.ipfire.org/?p=ipfire-2.x.git;a=tree;f=config/rootfiles/core/164/filelists;h=c81a06294cd3527b9b68d1352d487bf1c8f95c9e;hb=refs/heads/core164
>> 
>>> The IDS page locking issue should be fixed and shipped with the final
>>> C164 release.
>>> 
>>> See:
>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=85b1d83b2a6fe2beb8169f3e810e915c4ad54036
>>> 
>>> Please report if you are affected again by this issue.
>> 
>> I'll test and report.
>> 
>> Best,
>> Matthias
>> 
>>> You safely can drop those folder and files from "/tmp".
>>> 
>>> Best regards,
>>> 
>>> -Stefan
>>>> Hi,
>>>> 
>>>> ..I'd like to have *two* problems with the IDS/IPS... ;-)
>>>> 
>>>> 
>>>> 1. After upgrading to Core164 yesterday I took a look at the IDS GUI
>>>> today and found it locked with a spinning wheel at "Ruleset update in
>>>> progress. Please wait until all operations have completed
>>>> successfully..."
>>>> 
>>>> Despite this message IPS seemed to work.
>>>> 
>>>> So I searched and found that the file "/tmp/ids_page_locked" still
>>>> existed. After deleting this file manually the web GUI opened,
>>>> everything seems normal.
>>>> 
>>>> Three activated rulesets were running, see attachment "01".
>>>> 
>>>> But the Web GUI stopped because of lines 234ff in 'ids.cgi':
>>>> 
>>>> ...
>>>> # Check if the page is locked, in this case, the ids_page_lock_file
>>>> exists.
>>>> if (-e $IDS::ids_page_lock_file) {
>>>>         # Lock the webpage and print notice about autoupgrade of the
>>>> ruleset
>>>>         # is in progess.
>>>>         &working_notice("$Lang::tr{'ids ruleset autoupdate in
>>>> progress'}");
>>>> 
>>>>         # Loop and check if the file still exists.
>>>>         while(-e $IDS::ids_page_lock_file) {
>>>>                 # Sleep for a second and re-check.
>>>>                 sleep 1;
>>>>         }
>>>> ...
>>>> 
>>>> I searched again and found that there are two subroutines in
>>>> '/var/ipfire/ids-functions.pl', lines 1920ff, which are responsible
>>>> for
>>>> locking and unlocking this file:
>>>> 
>>>> ...
>>>> ## Function to write the lock file for locking the WUI, while
>>>> ## the autoupdate script runs.
>>>> #
>>>> sub lock_ids_page() {
>>>>         # Call subfunction to create the file.
>>>>         &create_empty_file($ids_page_lock_file);
>>>> }
>>>> 
>>>> #
>>>> ## Function to release the lock of the WUI, again.
>>>> #
>>>> sub unlock_ids_page() {
>>>>         # Delete lock file.
>>>>         unlink($ids_page_lock_file);
>>>> }
>>>> ...
>>>> 
>>>> Somehow lines 103ff in "/usr/local/bin/update-ids-ruleset" didn't
>>>> work/weren't executed(?):
>>>> 
>>>> ...
>>>> # Lock the IDS page.
>>>> &IDS::lock_ids_page();
>>>> 
>>>> # The script has requested a lock, so set locket to "1".
>>>> $locked = "1";
>>>> 
>>>> # Grab the configured providers.
>>>> &General::readhasharray("$IDS::providers_settings_file",
>>>> \%providers);
>>>> 
>>>> # Loop through the array of available providers.
>>>> foreach my $id (keys %providers) {
>>>>    # Assign some nice variabled.
>>>>    my $provider = $providers{$id}[0];
>>>>    my $autoupdate_status = $providers{$id}[3];
>>>> 
>>>>    # Skip the provider if autoupdate is not enabled.
>>>>    next unless($autoupdate_status eq "enabled");
>>>> 
>>>>    # Call the download function and gather the new ruleset for the
>>>> current processed provider.
>>>>    if(&IDS::downloadruleset($provider)) {
>>>>       # Store error message for displaying in the WUI.
>>>>       &IDS::_store_error_message("$provider: $Lang::tr{'could not
>>>> download latest updates'}");
>>>> 
>>>>       # Unlock the IDS page.
>>>>       &IDS::unlock_ids_page();
>>>> 
>>>>       # Exit.
>>>>       exit 0;
>>>>    }
>>>> ...
>>>> 
>>>> Manual updates - yesterday - were ok, but the automatic updates
>>>> tonight
>>>> didn't unlink the lock file: IDS GUI stopped at the next call today.
>>>> 
>>>> ##########
>>>> 
>>>> 2. There still exist a directory '/tmp/isd_tmp' containing 'conf'
>>>> ('config' and 'map' files from emerging-threats) and a 'rules'-dir
>>>> containing all '*.rules'-files. Leftovers!? Can I safely delete these
>>>> dirs?
>>>> 
>>>> And: can anyone confirm these two problems - or reproduce?
>>>> 
>>>> Best,
>>>> Matthias
>>> 
>>> 
>> 
>