Re: Betatest Guardian 2.0

[Matthias Fischer <matthias.fischer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 13568 bytes --]

Hi,

On 28.07.2016 20:05, Stefan Schantl wrote:
> New test version (004) available.

First test: thanks - seems to work for me!

Once started, 'guardian' memory usage is at 14342 KB and stays there, no
matter what I do. I'll keep on testing...

Best,
Matthias

> http://people.ipfire.org/~stevee/guardian-2.0/
> 
> 
> Changelog: http://people.ipfire.org/~stevee/guardian-2.0/Changelog.txt
> 
> Installation is the same way than all previous versions.
> 
> Please do a lot of testing, I'm still lacking of feedback for 
> 
> * owncloud
> * proper handling of reconnections on red
> * detection of rotating the logfiles (logrotate)
> 
> As usual please provide your feedback on this list and report any bugs
> to our bugtracker.
> 
> Best regards,
> 
> -Stefan
>> Hello testers,
>> 
>> after a lot of code debugging I was able to determine
>> the reason of those memory leak.
>> 
>> It is the default behavior of not freeing used virtual memory again
>> after a thread has been stopped.
>> 
>> Guardian stops and restarts each worker thread on a reload and a
>> logrotate event.
>> 
>> I'll have to rework the corresponding code to solve this issue and
>> come
>> back after finished this.
>> 
>> Thanks for pointing this out,
>> 
>> -Stefan
>> > 
>> > Did anyone try to monitor the size of the log files that guardian
>> > is
>> > parsing as
>> > well? Could it be that every line that is read remains in memory?
>> > 
>> > This is just an idea...
>> > 
>> > Best,
>> > -Michael
>> > 
>> > On Sat, 2016-07-23 at 00:23 +0200, Matthias Fischer wrote:
>> > > 
>> > > 
>> > > Correction: in the meanwhile it jumped to 47890 KB, I don't know
>> > > why.
>> > > Logrotation?.
>> > > 
>> > > On 22.07.2016 22:28, Matthias Fischer wrote:
>> > > > 
>> > > > 
>> > > > Hi,
>> > > > 
>> > > > ...for the records...:
>> > > > 
>> > > > Since I switched "Loglevel" to OFF, memory usage stays at
>> > > > "14333
>> > > > KB" and
>> > > > didn't change/rise since then.
>> > > > 
>> > > > HTH,
>> > > > Matthias
>> > > > 
>> > > > On 21.07.2016 23:07, Matthias Fischer wrote:
>> > > > > 
>> > > > > 
>> > > > > Hi,
>> > > > > 
>> > > > > Sounds interesting.
>> > > > > 
>> > > > > So I thought I take a little test...
>> > > > > 
>> > > > > Initial RAM-Usage: 14334 KB
>> > > > > 
>> > > > > First I just switched logging, did nothing else:
>> > > > > 
>> > > > > syslog => file => 22726 KB
>> > > > > file => syslog => 31117 KB
>> > > > > syslog => file => 39507/47898 KB (RAM suddenly altered. Why?
>> > > > > No
>> > > > > idea.)
>> > > > > file => syslog => 56289 KB
>> > > > > 
>> > > > > Restarted through console:
>> > > > > 
>> > > > > root(a)ipfire: /var/log/guardian # guardianctrl restart
>> > > > > Stopping Guardian...
>> > > > > Starting Guardian...
>> > > > > Unable to continue: /usr/sbin/guardian is running
>> > > > >                   [ WARN ]
>> > > > > 
>> > > > > Hm?
>> > > > > 
>> > > > > Stopped through console, no output, 'guardian' not found
>> > > > > anymore,
>> > > > > neither in GUI nor through console:
>> > > > > 
>> > > > > root(a)ipfire: /var/log/guardian # ps ax | grep guardian
>> > > > >  6962 pts/1    S+     0:00 grep guardian
>> > > > > 
>> > > > > Started through console and we're exactly where we started
>> > > > > (14334 KB).
>> > > > > 
>> > > > > The same happens if I switch the 'Priority-level' or the
>> > > > > 'Firewall-
>> > > > > Action'.
>> > > > > 
>> > > > > Initial: 2
>> > > > > 2 => 3 => 22723 KB
>> > > > > 3 => 2 => 31112 KB
>> > > > > 
>> > > > > Firewall-Action:
>> > > > > Reject => Drop => 39501 KB
>> > > > > 
>> > > > > Stop => Start => 14334 KB
>> > > > > 
>> > > > > Interestingly, during MY (log-)switching, 'guardian' never
>> > > > > stopped.
>> > > > > 
>> > > > > HTH,
>> > > > > Matthias
>> > > > > 
>> > > > > On 21.07.2016 21:52, Flying Trashcan wrote:
>> > > > > > 
>> > > > > > 
>> > > > > > I am now noticing that when I switch from Log facility
>> > > > > > “file”
>> > > > > > to
>> > > > > > “syslog”, Guardian Daemon stops and doesn’t
>> > > > > > restart.  Switching from
>> > > > > > syslog to file didn’t stop the service, only switching back
>> > > > > > to syslog
>> > > > > > from file.  I can manually start the service and be back to
>> > > > > > normal.  Not
>> > > > > > a big deal, but if someone made the switch and didn’t think
>> > > > > > to manually
>> > > > > > start the service, it could be left without running
>> > > > > > Guardian.
>> > > > > > 
>> > > > > > 
>> > > > > > 
>> > > > > > 
>> > > > > > > 
>> > > > > > > 
>> > > > > > > On Jul 21, 2016, at 4:25 AM, Matthias Fischer
>> > > > > > > <matthias.fis
>> > > > > > > cher(a)ipfire
>> > > > > > > .org> wrote:
>> > > > > > > 
>> > > > > > > Hi,
>> > > > > > > 
>> > > > > > > I mentioned this earlier, but it seems that 'guardian'
>> > > > > > > has
>> > > > > > > some kind
>> > > > > > > of
>> > > > > > > memory leak?
>> > > > > > > 
>> > > > > > > It started about two days ago with ~14 MB RAM. Then it
>> > > > > > > jumped to ~34
>> > > > > > > MB,
>> > > > > > > then to ~48 MB - today it suddenly uses 71 MB.
>> > > > > > > 
>> > > > > > > And if I start it on my testmachine (offline!) it uses
>> > > > > > > ~90
>> > > > > > > MB.
>> > > > > > > 
>> > > > > > > Can someone confirm?
>> > > > > > > 
>> > > > > > > Besides this, its working without seen problems.
>> > > > > > > 
>> > > > > > > Best,
>> > > > > > > Matthias
>> > > > > > > 
>> > > > > > > On 20.07.2016 15:33, Stefan Schantl wrote:
>> > > > > > > > 
>> > > > > > > > 
>> > > > > > > > Hello testers,
>> > > > > > > > 
>> > > > > > > > I've uploaded  a new test version (003).
>> > > > > > > > 
>> > > > > > > > Update or fresh install works like described in the
>> > > > > > > > announcement
>> > > > > > > > mail.
>> > > > > > > > 
>> > > > > > > > The Changelog can be found here:
>> > > > > > > > 
>> > > > > > > > http://people.ipfire.org/~stevee/guardian-2.0/Changelog
>> > > > > > > > .t
>> > > > > > > > xt
>> > > > > > > > 
>> > > > > > > > At the moment I'm missing feedback for the following
>> > > > > > > > functions:
>> > > > > > > > 
>> > > > > > > > * Manually blocking / unblocking addresses.
>> > > > > > > > * Dealing with the ignore list.
>> > > > > > > > * Owncloud message parser.
>> > > > > > > > * Logrotate, there should be an corresponding log entry
>> > > > > > > > in the
>> > > > > > > > guardian
>> > > > > > > > logfile after rotation of the logfiles have been done.
>> > > > > > > > * Reload of the ignore list after "Red" has been
>> > > > > > > > reconnected. There
>> > > > > > > > also a corresponding log entry should be logged to the
>> > > > > > > > logfile and
>> > > > > > > > the
>> > > > > > > > new "Red-address" should also be logged as part of the
>> > > > > > > > ignore list
>> > > > > > > > (If
>> > > > > > > > you own an dynamic assigned one).
>> > > > > > > > 
>> > > > > > > > As always please report your bugs or experience with
>> > > > > > > > the
>> > > > > > > > new version
>> > > > > > > > to
>> > > > > > > > this list.
>> > > > > > > > 
>> > > > > > > > Best regards,
>> > > > > > > > 
>> > > > > > > > -Stefan
>> > > > > > > > 
>> > > > > > > > > 
>> > > > > > > > > 
>> > > > > > > > > Hello mailing list followers,
>> > > > > > > > > 
>> > > > > > > > > this is the official release announcement for the
>> > > > > > > > > first
>> > > > > > > > > beta
>> > > > > > > > > release
>> > > > > > > > > of
>> > > > > > > > > the new Guardian 2.0 approach.
>> > > > > > > > > 
>> > > > > > > > > 
>> > > > > > > > > - What are the differences to the current version of
>> > > > > > > > > guardian
>> > > > > > > > > (legacy)
>> > > > > > > > > and the first approach of guardian 2.0?
>> > > > > > > > > 
>> > > > > > > > > The most important difference is, that the new
>> > > > > > > > > version
>> > > > > > > > > of Guardian
>> > > > > > > > > 2.0
>> > > > > > > > > completely has been re-written from scratch and
>> > > > > > > > > released under the
>> > > > > > > > > terms of the GPLv3. The legacy version of guardian is
>> > > > > > > > > not
>> > > > > > > > > maintained
>> > > > > > > > > anymore by it's developer and the software has been
>> > > > > > > > > released
>> > > > > > > > > without
>> > > > > > > > > any license details at all.
>> > > > > > > > > 
>> > > > > > > > > Guardian 2.0 has a very modular code base and has
>> > > > > > > > > been
>> > > > > > > > > designed as
>> > > > > > > > > a
>> > > > > > > > > multi-threaded application. This allows a parallel
>> > > > > > > > > parsing of all
>> > > > > > > > > monitored logfiles and faster actions, if one of the
>> > > > > > > > > used modules
>> > > > > > > > > detects an attack.
>> > > > > > > > > 
>> > > > > > > > > A very important difference to the legacy version is
>> > > > > > > > > the support
>> > > > > > > > > of
>> > > > > > > > > configuring and managing the entire service through
>> > > > > > > > > the
>> > > > > > > > > IPFire
>> > > > > > > > > webinterface. The entire configuration, managing of
>> > > > > > > > > current
>> > > > > > > > > blocked
>> > > > > > > > > hosts, unblocking them or editing the ignored hosts
>> > > > > > > > > list now can
>> > > > > > > > > be
>> > > > > > > > > done in a graphical way. 
>> > > > > > > > > 
>> > > > > > > > > The legacy version of guardian only supported parsing
>> > > > > > > > > snort
>> > > > > > > > > alerts.
>> > > > > > > > > HTTPD and SSH support has been patched by the IPFire
>> > > > > > > > > development
>> > > > > > > > > team
>> > > > > > > > > some time ago. Guardian 2.0 supports all of them out
>> > > > > > > > > of
>> > > > > > > > > the box
>> > > > > > > > > and
>> > > > > > > > > includes a filter to detect owncloud login brute-
>> > > > > > > > > force
>> > > > > > > > > attempts.
>> > > > > > > > > As a
>> > > > > > > > > benefit of the new modular design, additional filters
>> > > > > > > > > easily can
>> > > > > > > > > be
>> > > > > > > > > added.
>> > > > > > > > > 
>> > > > > > > > > Guardian 2.0 is able to reload it's configuration,
>> > > > > > > > > reloading
>> > > > > > > > > the ignore list during runtime and handle, if the
>> > > > > > > > > logfiles will
>> > > > > > > > > get
>> > > > > > > > > rotated by logrotate. This actions can be called by
>> > > > > > > > > using the
>> > > > > > > > > webinterface or from the command line interface by
>> > > > > > > > > using
>> > > > > > > > > "guardianctrl".
>> > > > > > > > > 
>> > > > > > > > > These are just a handful of the changes and benefits
>> > > > > > > > > which comes
>> > > > > > > > > with
>> > > > > > > > > Guardian 2.0, a complete list would be to long for
>> > > > > > > > > this
>> > > > > > > > > mailing
>> > > > > > > > > list.
>> > > > > > > > > 
>> > > > > > > > > 
>> > > > > > > > > - How to join testing?
>> > > > > > > > > 
>> > > > > > > > > To get part of the testing team, simple navigate to
>> > > > > > > > > htt
>> > > > > > > > > p://people.
>> > > > > > > > > ipf
>> > > > > > > > > ir
>> > > > > > > > > e.org/~stevee/guardian-2.0/ and download the latest
>> > > > > > > > > tarball
>> > > > > > > > > (currently
>> > > > > > > > > 002). Please take care to download the correct one,
>> > > > > > > > > based on your
>> > > > > > > > > used
>> > > > > > > > > architecture. The i585 packages are for 32Bit
>> > > > > > > > > installations of
>> > > > > > > > > IPFire,
>> > > > > > > > > the x86_64 packages only can be used on 64Bit
>> > > > > > > > > installations.
>> > > > > > > > > 
>> > > > > > > > > Put the downloaded file on your IPFire test system
>> > > > > > > > > and
>> > > > > > > > > extract the
>> > > > > > > > > package by using "tar -xvf guardian-2.0-
>> > > > > > > > > 002.<arch>.tar.gz -C /".
>> > > > > > > > > 
>> > > > > > > > > The final installation step would be to regenerate
>> > > > > > > > > the
>> > > > > > > > > language
>> > > > > > > > > cache
>> > > > > > > > > by executing "update-lang-cache" on the console.
>> > > > > > > > > 
>> > > > > > > > > From now you can find a new menu item called
>> > > > > > > > > "Guardian"
>> > > > > > > > > in your
>> > > > > > > > > "Service" menu after you have logged-in into your
>> > > > > > > > > IPFire's
>> > > > > > > > > webinterface.
>> > > > > > > > > 
>> > > > > > > > > Documentation can be found on the IPFire wiki:
>> > > > > > > > > http://w
>> > > > > > > > > iki.ipfire.
>> > > > > > > > > org
>> > > > > > > > > /e
>> > > > > > > > > n/addons/guardian/start#the_guardian_20_addon
>> > > > > > > > > 
>> > > > > > > > > 
>> > > > > > > > > - Where to post bugs reports or provide feedback?
>> > > > > > > > > 
>> > > > > > > > > If you find any bugs, please report them as usual on
>> > > > > > > > > the IPFire
>> > > > > > > > > bugtracker, which can be found at https://bugzilla.ip
>> > > > > > > > > fi
>> > > > > > > > > re.org.
>> > > > > > > > > 
>> > > > > > > > > To provide feedback or to join a discussion, please
>> > > > > > > > > send your
>> > > > > > > > > mails
>> > > > > > > > > to
>> > > > > > > > > "development(a)lists.ipfire.org" (Please register first
>> > > > > > > > > at http://li
>> > > > > > > > > sts
>> > > > > > > > > .i
>> > > > > > > > > pfire.org if not yet done).
>> > > > > > > > > 
>> > > > > > > > > The source code can be found at http://git.ipfire.org
>> > > > > > > > > /?
>> > > > > > > > > p=people/st
>> > > > > > > > > eve
>> > > > > > > > > e/
>> > > > > > > > > guardian.git;a=summary
>> > > > > > > > > 
>> > > > > > > > > 
>> > > > > > > > > Happy testing,
>> > > > > > > > > 
>> > > > > > > > > -Stefan
>> > > > > > > > > 
>