Re: Betatest Guardian 2.0

[Stefan Schantl <stefan.schantl@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 11327 bytes --]

Hello testers,

after a lot of code debugging I was able to determine
the reason of those memory leak.

It is the default behavior of not freeing used virtual memory again
after a thread has been stopped.

Guardian stops and restarts each worker thread on a reload and a
logrotate event.

I'll have to rework the corresponding code to solve this issue and come
back after finished this.

Thanks for pointing this out,

-Stefan
> Did anyone try to monitor the size of the log files that guardian is
> parsing as
> well? Could it be that every line that is read remains in memory?
> 
> This is just an idea...
> 
> Best,
> -Michael
> 
> On Sat, 2016-07-23 at 00:23 +0200, Matthias Fischer wrote:
> > 
> > Correction: in the meanwhile it jumped to 47890 KB, I don't know
> > why.
> > Logrotation?.
> > 
> > On 22.07.2016 22:28, Matthias Fischer wrote:
> > > 
> > > Hi,
> > > 
> > > ...for the records...:
> > > 
> > > Since I switched "Loglevel" to OFF, memory usage stays at "14333
> > > KB" and
> > > didn't change/rise since then.
> > > 
> > > HTH,
> > > Matthias
> > > 
> > > On 21.07.2016 23:07, Matthias Fischer wrote:
> > > > 
> > > > Hi,
> > > > 
> > > > Sounds interesting.
> > > > 
> > > > So I thought I take a little test...
> > > > 
> > > > Initial RAM-Usage: 14334 KB
> > > > 
> > > > First I just switched logging, did nothing else:
> > > > 
> > > > syslog => file => 22726 KB
> > > > file => syslog => 31117 KB
> > > > syslog => file => 39507/47898 KB (RAM suddenly altered. Why? No
> > > > idea.)
> > > > file => syslog => 56289 KB
> > > > 
> > > > Restarted through console:
> > > > 
> > > > root(a)ipfire: /var/log/guardian # guardianctrl restart
> > > > Stopping Guardian...
> > > > Starting Guardian...
> > > > Unable to continue: /usr/sbin/guardian is running
> > > >                   [ WARN ]
> > > > 
> > > > Hm?
> > > > 
> > > > Stopped through console, no output, 'guardian' not found
> > > > anymore,
> > > > neither in GUI nor through console:
> > > > 
> > > > root(a)ipfire: /var/log/guardian # ps ax | grep guardian
> > > >  6962 pts/1    S+     0:00 grep guardian
> > > > 
> > > > Started through console and we're exactly where we started
> > > > (14334 KB).
> > > > 
> > > > The same happens if I switch the 'Priority-level' or the
> > > > 'Firewall-
> > > > Action'.
> > > > 
> > > > Initial: 2
> > > > 2 => 3 => 22723 KB
> > > > 3 => 2 => 31112 KB
> > > > 
> > > > Firewall-Action:
> > > > Reject => Drop => 39501 KB
> > > > 
> > > > Stop => Start => 14334 KB
> > > > 
> > > > Interestingly, during MY (log-)switching, 'guardian' never
> > > > stopped.
> > > > 
> > > > HTH,
> > > > Matthias
> > > > 
> > > > On 21.07.2016 21:52, Flying Trashcan wrote:
> > > > > 
> > > > > I am now noticing that when I switch from Log facility “file”
> > > > > to
> > > > > “syslog”, Guardian Daemon stops and doesn’t
> > > > > restart.  Switching from
> > > > > syslog to file didn’t stop the service, only switching back
> > > > > to syslog
> > > > > from file.  I can manually start the service and be back to
> > > > > normal.  Not
> > > > > a big deal, but if someone made the switch and didn’t think
> > > > > to manually
> > > > > start the service, it could be left without running Guardian.
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > > 
> > > > > > On Jul 21, 2016, at 4:25 AM, Matthias Fischer <matthias.fis
> > > > > > cher(a)ipfire
> > > > > > .org> wrote:
> > > > > > 
> > > > > > Hi,
> > > > > > 
> > > > > > I mentioned this earlier, but it seems that 'guardian' has
> > > > > > some kind
> > > > > > of
> > > > > > memory leak?
> > > > > > 
> > > > > > It started about two days ago with ~14 MB RAM. Then it
> > > > > > jumped to ~34
> > > > > > MB,
> > > > > > then to ~48 MB - today it suddenly uses 71 MB.
> > > > > > 
> > > > > > And if I start it on my testmachine (offline!) it uses ~90
> > > > > > MB.
> > > > > > 
> > > > > > Can someone confirm?
> > > > > > 
> > > > > > Besides this, its working without seen problems.
> > > > > > 
> > > > > > Best,
> > > > > > Matthias
> > > > > > 
> > > > > > On 20.07.2016 15:33, Stefan Schantl wrote:
> > > > > > > 
> > > > > > > Hello testers,
> > > > > > > 
> > > > > > > I've uploaded  a new test version (003).
> > > > > > > 
> > > > > > > Update or fresh install works like described in the
> > > > > > > announcement
> > > > > > > mail.
> > > > > > > 
> > > > > > > The Changelog can be found here:
> > > > > > > 
> > > > > > > http://people.ipfire.org/~stevee/guardian-2.0/Changelog.t
> > > > > > > xt
> > > > > > > 
> > > > > > > At the moment I'm missing feedback for the following
> > > > > > > functions:
> > > > > > > 
> > > > > > > * Manually blocking / unblocking addresses.
> > > > > > > * Dealing with the ignore list.
> > > > > > > * Owncloud message parser.
> > > > > > > * Logrotate, there should be an corresponding log entry
> > > > > > > in the
> > > > > > > guardian
> > > > > > > logfile after rotation of the logfiles have been done.
> > > > > > > * Reload of the ignore list after "Red" has been
> > > > > > > reconnected. There
> > > > > > > also a corresponding log entry should be logged to the
> > > > > > > logfile and
> > > > > > > the
> > > > > > > new "Red-address" should also be logged as part of the
> > > > > > > ignore list
> > > > > > > (If
> > > > > > > you own an dynamic assigned one).
> > > > > > > 
> > > > > > > As always please report your bugs or experience with the
> > > > > > > new version
> > > > > > > to
> > > > > > > this list.
> > > > > > > 
> > > > > > > Best regards,
> > > > > > > 
> > > > > > > -Stefan
> > > > > > > 
> > > > > > > > 
> > > > > > > > Hello mailing list followers,
> > > > > > > > 
> > > > > > > > this is the official release announcement for the first
> > > > > > > > beta
> > > > > > > > release
> > > > > > > > of
> > > > > > > > the new Guardian 2.0 approach.
> > > > > > > > 
> > > > > > > > 
> > > > > > > > - What are the differences to the current version of
> > > > > > > > guardian
> > > > > > > > (legacy)
> > > > > > > > and the first approach of guardian 2.0?
> > > > > > > > 
> > > > > > > > The most important difference is, that the new version
> > > > > > > > of Guardian
> > > > > > > > 2.0
> > > > > > > > completely has been re-written from scratch and
> > > > > > > > released under the
> > > > > > > > terms of the GPLv3. The legacy version of guardian is
> > > > > > > > not
> > > > > > > > maintained
> > > > > > > > anymore by it's developer and the software has been
> > > > > > > > released
> > > > > > > > without
> > > > > > > > any license details at all.
> > > > > > > > 
> > > > > > > > Guardian 2.0 has a very modular code base and has been
> > > > > > > > designed as
> > > > > > > > a
> > > > > > > > multi-threaded application. This allows a parallel
> > > > > > > > parsing of all
> > > > > > > > monitored logfiles and faster actions, if one of the
> > > > > > > > used modules
> > > > > > > > detects an attack.
> > > > > > > > 
> > > > > > > > A very important difference to the legacy version is
> > > > > > > > the support
> > > > > > > > of
> > > > > > > > configuring and managing the entire service through the
> > > > > > > > IPFire
> > > > > > > > webinterface. The entire configuration, managing of
> > > > > > > > current
> > > > > > > > blocked
> > > > > > > > hosts, unblocking them or editing the ignored hosts
> > > > > > > > list now can
> > > > > > > > be
> > > > > > > > done in a graphical way. 
> > > > > > > > 
> > > > > > > > The legacy version of guardian only supported parsing
> > > > > > > > snort
> > > > > > > > alerts.
> > > > > > > > HTTPD and SSH support has been patched by the IPFire
> > > > > > > > development
> > > > > > > > team
> > > > > > > > some time ago. Guardian 2.0 supports all of them out of
> > > > > > > > the box
> > > > > > > > and
> > > > > > > > includes a filter to detect owncloud login brute-force
> > > > > > > > attempts.
> > > > > > > > As a
> > > > > > > > benefit of the new modular design, additional filters
> > > > > > > > easily can
> > > > > > > > be
> > > > > > > > added.
> > > > > > > > 
> > > > > > > > Guardian 2.0 is able to reload it's configuration,
> > > > > > > > reloading
> > > > > > > > the ignore list during runtime and handle, if the
> > > > > > > > logfiles will
> > > > > > > > get
> > > > > > > > rotated by logrotate. This actions can be called by
> > > > > > > > using the
> > > > > > > > webinterface or from the command line interface by
> > > > > > > > using
> > > > > > > > "guardianctrl".
> > > > > > > > 
> > > > > > > > These are just a handful of the changes and benefits
> > > > > > > > which comes
> > > > > > > > with
> > > > > > > > Guardian 2.0, a complete list would be to long for this
> > > > > > > > mailing
> > > > > > > > list.
> > > > > > > > 
> > > > > > > > 
> > > > > > > > - How to join testing?
> > > > > > > > 
> > > > > > > > To get part of the testing team, simple navigate to htt
> > > > > > > > p://people.
> > > > > > > > ipf
> > > > > > > > ir
> > > > > > > > e.org/~stevee/guardian-2.0/ and download the latest
> > > > > > > > tarball
> > > > > > > > (currently
> > > > > > > > 002). Please take care to download the correct one,
> > > > > > > > based on your
> > > > > > > > used
> > > > > > > > architecture. The i585 packages are for 32Bit
> > > > > > > > installations of
> > > > > > > > IPFire,
> > > > > > > > the x86_64 packages only can be used on 64Bit
> > > > > > > > installations.
> > > > > > > > 
> > > > > > > > Put the downloaded file on your IPFire test system and
> > > > > > > > extract the
> > > > > > > > package by using "tar -xvf guardian-2.0-
> > > > > > > > 002.<arch>.tar.gz -C /".
> > > > > > > > 
> > > > > > > > The final installation step would be to regenerate the
> > > > > > > > language
> > > > > > > > cache
> > > > > > > > by executing "update-lang-cache" on the console.
> > > > > > > > 
> > > > > > > > From now you can find a new menu item called "Guardian"
> > > > > > > > in your
> > > > > > > > "Service" menu after you have logged-in into your
> > > > > > > > IPFire's
> > > > > > > > webinterface.
> > > > > > > > 
> > > > > > > > Documentation can be found on the IPFire wiki: http://w
> > > > > > > > iki.ipfire.
> > > > > > > > org
> > > > > > > > /e
> > > > > > > > n/addons/guardian/start#the_guardian_20_addon
> > > > > > > > 
> > > > > > > > 
> > > > > > > > - Where to post bugs reports or provide feedback?
> > > > > > > > 
> > > > > > > > If you find any bugs, please report them as usual on
> > > > > > > > the IPFire
> > > > > > > > bugtracker, which can be found at https://bugzilla.ipfi
> > > > > > > > re.org.
> > > > > > > > 
> > > > > > > > To provide feedback or to join a discussion, please
> > > > > > > > send your
> > > > > > > > mails
> > > > > > > > to
> > > > > > > > "development(a)lists.ipfire.org" (Please register first
> > > > > > > > at http://li
> > > > > > > > sts
> > > > > > > > .i
> > > > > > > > pfire.org if not yet done).
> > > > > > > > 
> > > > > > > > The source code can be found at http://git.ipfire.org/?
> > > > > > > > p=people/st
> > > > > > > > eve
> > > > > > > > e/
> > > > > > > > guardian.git;a=summary
> > > > > > > > 
> > > > > > > > 
> > > > > > > > Happy testing,
> > > > > > > > 
> > > > > > > > -Stefan
> > > > > > > > 
> > > > > 
> > > > 
> > > 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]