[SPAM Warning!]Re: Zabbix support for suricata-reporter

[Robin Roevens <robin.roevens@disroot.org>]

Hi Michael

Michael Tremer schreef op za 04-10-2025 om 11:52 [+0100]:
> Hello Robin,
> 
> > On 2 Oct 2025, at 22:04, Robin Roevens <robin.roevens@disroot.org>
> > wrote:
> > 
> > Hi Michael
> > 
> > I saw there is a suricata-reporter in the upcoming CU. And I was
> > wondering if I could add an additional reporter into it for sending
> > alerts straight to Zabbix, next to syslog and email.
> 
> I think that is a great idea!
That makes at least 2 of us :-)

> 
> > I have already been experimenting with parsing fast.log using the
> > zabbix_agentd, which seems to work quite well. But since there is
> > now a
> > reporter, it would be nice to have it support sending alerts to
> > zabbix
> > directly instead of zabbix separately monitoring the fast.log file.
> 
> Parsing log files like that is never fun. I also think that I
> remember that suricata has deprecated support for writing fast.log,
> but then it does not seem to be deprecated any more. Looking at the
> alternative EVE format, fast.log is just there to have something that
> is somewhat human-readable, but at the same time does not carry a lot
> of information.
I had considered EVE logs but it was not really clear to me how to
distinguish normal events and actual alert events from than, and I saw
that it is currently disabled, but that the reporter will soon enable
it to connect to a socket. So I won't be able to listen in with the
zabbix_agent.. I think.. 
Anyway, no longer relevant if it is build in into the reporter.
 
> 
> > If that would be ok for you. There are 2 possible ways to do this:
> > - using the zabbix_utils python library:
> > https://blog.zabbix.com/python-zabbix-utils/27056/
> > - or using the zabbix_sender command utility that currently gets
> > installed when installing zabbix_agentd
> 
> The IPS could generate a lot of alerts. Especially when there is
> bursts in (malicious) traffic and I am already unhappy with calling
> sendmail - but there I have no other choice.
> 
> The daemon initially had some worker daemons so that reading from
> Suricata would be entirely decoupled from processing any messages.
> However, Python’s multiprocessing capabilities are really bad and so
> now it is a threaded implementation. Python isn’t good at that
> either, but when we are calling a subprocess the GIL is being
> released and so we should not actually spend too much time in the
> Python code itself.
> 
> This has to be considered when adding yet another output so that the
> process does not get stalled when Zabbix does not respond, is
> overwhelmed or anything else. We could simply combat that by adding a
> couple more worker threads if necessary - but I am getting ahead of
> myself.
> 
> So the Python module would be by far preferable. It seems to have a
> nice API, but we will have to check how it works underneath to reach
> the objectives outlined above.
By default the zabbix sender functionality of the library has a timeout
of 10s. When the Zabbix server is healthy, that timeout is never
reached. But when unavailable or overloaded, that can happen. This
timeout is configurable.
It also has the possibility to bulk-send entries, which it will
internally split into chunks of max 250 items which will then each be
sent separately. Each chunk then has the 10s timout. Chunksize can be
configured also.
So to minimize zabbix communications, it could be a possibility to
cache events within a timeframe (say 30s) and only once per 30s send
all events in the queue. 30s is no longer 'real-time', so I would like
to avoid that, but it may be a good idea to at least buffer within 1s
to prevent possible hammering the server every millisecond in some
extreme cases?

There is also an async version of the sender functionality included in
the library. I've not yet done any async python code, so I'm not sure I
understand it correctly, but I assume it can make the actual sending a
sort of background job, while the reporter can continue its other work?

> 
> > I assume, using the python library will probably be the most
> > performant
> > option; But then I should also create a zabbix_utils python library
> > pak-file?
> 
> Hmm, this is where it is getting interesting.
> 
> The suricata-reporter stuff is supposed to be rather distro agnostic.
> Besides Python itself, it only has a dependency to sqlite3 (which
> normally comes with the Python standard library) and reportlab to
> generate the PDFs. The goal is to run it on IPFire 2 for now, but it
> would also be pretty much ready as is to run in IPFire 3 some day. Of
> course it runs on any other distro - although I don’t know what it
> would do on a non-Linux OS.
> 
> I would like to keep it lightweight and hopefully some more people
> would adopt it.
> 
> I believe adding zabbix should be done as an optional dependency. We
> could simply try to import the Zabbix modules. If they are there, the
> process has support for Zabbix. If not, then not. That could simply
> be handled at runtime.
> 
> That way, we could have zabbix as an add-on in IPFire instead of
> adding it to the core distribution and if other people want to run
> the whole thing, they can enjoy a minimum amount of dependencies and
> don’t have to build/install any zabbix packages if they don’t need
> them.

I agree that suricata-reporter in itself should be as lightweight and
distro agnostic as possible. Adding zabbix as an optional dependency is
an option. But I was wondering if it wouldn't be more elegant to have
suricata-reporter support some sort of reporter modules or plugins,
which could then contain the email, pdf and zabbix functionalities.
That way, you could ship suricata-reporter without zabbix
functionalities, and I could create a suricata-reporter-zabbix-plugin
with the zabbix specific code in it that, when installed, is picked up
by suricata-reporter. For IPFire I could then just include the reporter
plugin in the zabbix_agentd pak on in a dependent separate pak which in
turn depends on a new zabbix-utils pak containing the zabbix python
library, and for other distro's, it can be downloaded from my git,
and/or I add it to an ipfire git repo?

> 
> > Both the python module and the commandline cli have the possibility
> > to
> > get zabbix server connection info from the zabbix_agentd configfile
> > so
> > config of the reporter would be something like:
> > [zabbix]
> > enabled = true
> > zabbix_agentd_config = /etc/zabbix_agentd/zabbix_agentd.conf
> > alert_item_key = ipfire.suricata.event.get
> 
> I mean, this is all up to you. I would definitely do this bit:
> 
> [zabbix]
> enabled = true
> 
> That way it is consistent with the remainder. Any other options, well
> I don’t know what you would need :) I do enjoy a daemon that does not
> need a lot of configuration though and therefore can be run with only
> a very small configuration as it relies on sensible defaults.
I think what I gave as an example, could suffice. It may be usefull to
support also explicit zabbix server settings instead of referencing the
zabbix_agentd config. On IPFire, using the zabbix_agentd config is
probably the best option to avoid duplicate configurations. On other
distro's or use cases of the reporter, it may well be that it is used
without a zabbix_agent installed, hence without a pre-existing config
that can be used to get zabbix server details from.

And in case of the plugin-architecture, I think easiest is to also
support a loadable config-directory, so config files from other pak-
files can be dropped in it?

> 
> > Then the reporter can format the incoming suricata alert/event as
> > json
> > and send it to the configured alert_item_key on the zabbix server
> > as
> > configured in the zabbix_agentd.conf
> 
> We do store the entire JSON object as it comes from Suricata in the
> sqlite3 database. That way we have all the information that could
> possible be available to us even though we don’t need everything
> right now. The fast.log format definitely lacks a lot of important
> information, but it is nicely readable.
> 
> Depending on what processing Zabbix can do later and what information
> it permanently-ish stores, I would send the whole lot. The JSON
> objects are small enough and over the wire they would compress
> nicely, too. On the daemon side I wouldn’t implement this to be
> configurable because that only makes everything more complicated.

I will have to study what would exactly come through. But zabbix has a
quite powerfull preprocessing engine which can format, validate, split,
and can do almost anything with the incoming data. So that would
probably indeed be the easiest method

> 
> > Is this something you are open to? Then I can try to create a patch
> > for
> > suricata-reporter. (where should I then submit it? Also on this
> > list?)
> 
> Yes, very much so.
> 
> Patches can go on this list.
> 
> Let’s talk about any implementation details in advance. I feel a
> little bit precious about this program because it is nice, shiny and
> new and the code is very simple. I would like to keep it that way.
And here I go suggesting wild extensible plugin frameworks :-)
However, I don't think that should be very complicated. And it would
create an ability to keep the reporter clean, and leave possible future
extra plugins in the hands of others.
 
> 
> > If not I will have to continue working on the fast.log parsing.
> 
> Don’t. That will only make you unhappy.
Thanks.. I need some happiness once in a while :-)
> 
> > And while on the topic of monitoring suricata; I would like to get
> > some
> > extra stats from it, which, for as far as I currently know, can be
> > retrieved using the suricata unix-socket that is currently disabled
> > by
> > default on ipfire. Many seem to use a 'suricatasc' tool to query
> > suricata using that socket, but that tool is not available on
> > ipfire.
> 
> What stats do you want/need?
For starters I was thinking iface-list and iface-stat to get interface-
specific stats about number of packets scanned, dropped, bypassed. But
I will have to experiment a bit with it to see what stats can be
valuable.
> 
> I am happy to turn on the socket and built the tool. Does it come
> with the suricata package?
I think it is a build option. I saw Adolf already proposed to enable
it.

> 
> Please submit patches :) I think this could be done first before
> diving into Zabbix because it should be a very simple change.
> 
> > Is it possible to have it on ipfire?, or should I start
> > experimenting
> > using socat?
> 
> No, if there are tools built already, use the code that is available.
ok

> 
> > And if succesful, is it then allowed for a future zabbix_agentd
> > addon
> > pak to enable that socket in the suricata config?
> 
> Enable the socket by default. It won’t hurt anyone.
ok.. most certainly it is better to have it enabled by default when the
suricatasc tool is also available on IPFire. Giving users also the
possibility to use it manually or to query suricatta using other tools
than Zabbix.

> 
> > If you dislike the idea of enabling and querying the socket,
> > another
> > possibility is having suricata dump stats in a seperate stats.log
> > which
> > I should then be able to parse using Zabbix.
> 
> Hmm, but then we are back to parsing logs. That is never fun.
agreed
> 
> > Before I start any implementations, what are your thoughts about
> > all
> > this ?
> 
> Very very happy with this here.
> 
> Let me know how I can help.
> 
> -Michael
> 
> > Regards
> > Robin
> > 
> > -- 
> > Dit bericht is gescanned op virussen en andere gevaarlijke
> > inhoud door MailScanner en lijkt schoon te zijn.
> > 
> 
Regards
Robin

-- 
Dit bericht is gescanned op virussen en andere gevaarlijke
inhoud door MailScanner en lijkt schoon te zijn.