Zabbix support for suricata-reporter

Zabbix support for suricata-reporter

[Robin Roevens]

Hi Michael

I saw there is a suricata-reporter in the upcoming CU. And I was
wondering if I could add an additional reporter into it for sending
alerts straight to Zabbix, next to syslog and email.
I have already been experimenting with parsing fast.log using the
zabbix_agentd, which seems to work quite well. But since there is now a
reporter, it would be nice to have it support sending alerts to zabbix
directly instead of zabbix separately monitoring the fast.log file.

If that would be ok for you. There are 2 possible ways to do this:
- using the zabbix_utils python library:
https://blog.zabbix.com/python-zabbix-utils/27056/
- or using the zabbix_sender command utility that currently gets
installed when installing zabbix_agentd

I assume, using the python library will probably be the most performant
option; But then I should also create a zabbix_utils python library
pak-file?

Both the python module and the commandline cli have the possibility to
get zabbix server connection info from the zabbix_agentd configfile so
config of the reporter would be something like:
[zabbix]
enabled = true
zabbix_agentd_config = /etc/zabbix_agentd/zabbix_agentd.conf
alert_item_key = ipfire.suricata.event.get

Then the reporter can format the incoming suricata alert/event as json
and send it to the configured alert_item_key on the zabbix server as
configured in the zabbix_agentd.conf

Is this something you are open to? Then I can try to create a patch for
suricata-reporter. (where should I then submit it? Also on this list?)

If not I will have to continue working on the fast.log parsing.

And while on the topic of monitoring suricata; I would like to get some
extra stats from it, which, for as far as I currently know, can be
retrieved using the suricata unix-socket that is currently disabled by
default on ipfire. Many seem to use a 'suricatasc' tool to query
suricata using that socket, but that tool is not available on ipfire.
Is it possible to have it on ipfire?, or should I start experimenting
using socat? 
And if succesful, is it then allowed for a future zabbix_agentd addon
pak to enable that socket in the suricata config?
If you dislike the idea of enabling and querying the socket, another
possibility is having suricata dump stats in a seperate stats.log which
I should then be able to parse using Zabbix.

Before I start any implementations, what are your thoughts about all
this ?

Regards
Robin

-- 
Dit bericht is gescanned op virussen en andere gevaarlijke
inhoud door MailScanner en lijkt schoon te zijn.

Re: Zabbix support for suricata-reporter

[Michael Tremer]

Hello Robin,

> On 2 Oct 2025, at 22:04, Robin Roevens <robin.roevens@disroot.org> wrote:
> 
> Hi Michael
> 
> I saw there is a suricata-reporter in the upcoming CU. And I was
> wondering if I could add an additional reporter into it for sending
> alerts straight to Zabbix, next to syslog and email.

I think that is a great idea!

> I have already been experimenting with parsing fast.log using the
> zabbix_agentd, which seems to work quite well. But since there is now a
> reporter, it would be nice to have it support sending alerts to zabbix
> directly instead of zabbix separately monitoring the fast.log file.

Parsing log files like that is never fun. I also think that I remember that suricata has deprecated support for writing fast.log, but then it does not seem to be deprecated any more. Looking at the alternative EVE format, fast.log is just there to have something that is somewhat human-readable, but at the same time does not carry a lot of information.

> If that would be ok for you. There are 2 possible ways to do this:
> - using the zabbix_utils python library:
> https://blog.zabbix.com/python-zabbix-utils/27056/
> - or using the zabbix_sender command utility that currently gets
> installed when installing zabbix_agentd

The IPS could generate a lot of alerts. Especially when there is bursts in (malicious) traffic and I am already unhappy with calling sendmail - but there I have no other choice.

The daemon initially had some worker daemons so that reading from Suricata would be entirely decoupled from processing any messages. However, Python’s multiprocessing capabilities are really bad and so now it is a threaded implementation. Python isn’t good at that either, but when we are calling a subprocess the GIL is being released and so we should not actually spend too much time in the Python code itself.

This has to be considered when adding yet another output so that the process does not get stalled when Zabbix does not respond, is overwhelmed or anything else. We could simply combat that by adding a couple more worker threads if necessary - but I am getting ahead of myself.

So the Python module would be by far preferable. It seems to have a nice API, but we will have to check how it works underneath to reach the objectives outlined above.

> I assume, using the python library will probably be the most performant
> option; But then I should also create a zabbix_utils python library
> pak-file?

Hmm, this is where it is getting interesting.

The suricata-reporter stuff is supposed to be rather distro agnostic. Besides Python itself, it only has a dependency to sqlite3 (which normally comes with the Python standard library) and reportlab to generate the PDFs. The goal is to run it on IPFire 2 for now, but it would also be pretty much ready as is to run in IPFire 3 some day. Of course it runs on any other distro - although I don’t know what it would do on a non-Linux OS.

I would like to keep it lightweight and hopefully some more people would adopt it.

I believe adding zabbix should be done as an optional dependency. We could simply try to import the Zabbix modules. If they are there, the process has support for Zabbix. If not, then not. That could simply be handled at runtime.

That way, we could have zabbix as an add-on in IPFire instead of adding it to the core distribution and if other people want to run the whole thing, they can enjoy a minimum amount of dependencies and don’t have to build/install any zabbix packages if they don’t need them.

> Both the python module and the commandline cli have the possibility to
> get zabbix server connection info from the zabbix_agentd configfile so
> config of the reporter would be something like:
> [zabbix]
> enabled = true
> zabbix_agentd_config = /etc/zabbix_agentd/zabbix_agentd.conf
> alert_item_key = ipfire.suricata.event.get

I mean, this is all up to you. I would definitely do this bit:

[zabbix]
enabled = true

That way it is consistent with the remainder. Any other options, well I don’t know what you would need :) I do enjoy a daemon that does not need a lot of configuration though and therefore can be run with only a very small configuration as it relies on sensible defaults.

> Then the reporter can format the incoming suricata alert/event as json
> and send it to the configured alert_item_key on the zabbix server as
> configured in the zabbix_agentd.conf

We do store the entire JSON object as it comes from Suricata in the sqlite3 database. That way we have all the information that could possible be available to us even though we don’t need everything right now. The fast.log format definitely lacks a lot of important information, but it is nicely readable.

Depending on what processing Zabbix can do later and what information it permanently-ish stores, I would send the whole lot. The JSON objects are small enough and over the wire they would compress nicely, too. On the daemon side I wouldn’t implement this to be configurable because that only makes everything more complicated.

> Is this something you are open to? Then I can try to create a patch for
> suricata-reporter. (where should I then submit it? Also on this list?)

Yes, very much so.

Patches can go on this list.

Let’s talk about any implementation details in advance. I feel a little bit precious about this program because it is nice, shiny and new and the code is very simple. I would like to keep it that way.

> If not I will have to continue working on the fast.log parsing.

Don’t. That will only make you unhappy.

> And while on the topic of monitoring suricata; I would like to get some
> extra stats from it, which, for as far as I currently know, can be
> retrieved using the suricata unix-socket that is currently disabled by
> default on ipfire. Many seem to use a 'suricatasc' tool to query
> suricata using that socket, but that tool is not available on ipfire.

What stats do you want/need?

I am happy to turn on the socket and built the tool. Does it come with the suricata package?

Please submit patches :) I think this could be done first before diving into Zabbix because it should be a very simple change.

> Is it possible to have it on ipfire?, or should I start experimenting
> using socat?

No, if there are tools built already, use the code that is available.

> And if succesful, is it then allowed for a future zabbix_agentd addon
> pak to enable that socket in the suricata config?

Enable the socket by default. It won’t hurt anyone.

> If you dislike the idea of enabling and querying the socket, another
> possibility is having suricata dump stats in a seperate stats.log which
> I should then be able to parse using Zabbix.

Hmm, but then we are back to parsing logs. That is never fun.

> Before I start any implementations, what are your thoughts about all
> this ?

Very very happy with this here.

Let me know how I can help.

-Michael

> Regards
> Robin
> 
> -- 
> Dit bericht is gescanned op virussen en andere gevaarlijke
> inhoud door MailScanner en lijkt schoon te zijn.
> 

Re: Zabbix support for suricata-reporter

[Adolf Belka]

Hi Robin,

On 04/10/2025 12:52, Michael Tremer wrote:
> Hello Robin,
> 
>> On 2 Oct 2025, at 22:04, Robin Roevens <robin.roevens@disroot.org> wrote:
>>
>> Hi Michael
>>
>> I saw there is a suricata-reporter in the upcoming CU. And I was
>> wondering if I could add an additional reporter into it for sending
>> alerts straight to Zabbix, next to syslog and email.
> 
> I think that is a great idea!
> 
>> I have already been experimenting with parsing fast.log using the
>> zabbix_agentd, which seems to work quite well. But since there is now a
>> reporter, it would be nice to have it support sending alerts to zabbix
>> directly instead of zabbix separately monitoring the fast.log file.
> 
> Parsing log files like that is never fun. I also think that I remember that suricata has deprecated support for writing fast.log, but then it does not seem to be deprecated any more. Looking at the alternative EVE format, fast.log is just there to have something that is somewhat human-readable, but at the same time does not carry a lot of information.
> 
>> If that would be ok for you. There are 2 possible ways to do this:
>> - using the zabbix_utils python library:
>> https://blog.zabbix.com/python-zabbix-utils/27056/
>> - or using the zabbix_sender command utility that currently gets
>> installed when installing zabbix_agentd
> 
> The IPS could generate a lot of alerts. Especially when there is bursts in (malicious) traffic and I am already unhappy with calling sendmail - but there I have no other choice.
> 
> The daemon initially had some worker daemons so that reading from Suricata would be entirely decoupled from processing any messages. However, Python’s multiprocessing capabilities are really bad and so now it is a threaded implementation. Python isn’t good at that either, but when we are calling a subprocess the GIL is being released and so we should not actually spend too much time in the Python code itself.
> 
> This has to be considered when adding yet another output so that the process does not get stalled when Zabbix does not respond, is overwhelmed or anything else. We could simply combat that by adding a couple more worker threads if necessary - but I am getting ahead of myself.
> 
> So the Python module would be by far preferable. It seems to have a nice API, but we will have to check how it works underneath to reach the objectives outlined above.
> 
>> I assume, using the python library will probably be the most performant
>> option; But then I should also create a zabbix_utils python library
>> pak-file?
> 
> Hmm, this is where it is getting interesting.
> 
> The suricata-reporter stuff is supposed to be rather distro agnostic. Besides Python itself, it only has a dependency to sqlite3 (which normally comes with the Python standard library) and reportlab to generate the PDFs. The goal is to run it on IPFire 2 for now, but it would also be pretty much ready as is to run in IPFire 3 some day. Of course it runs on any other distro - although I don’t know what it would do on a non-Linux OS.
> 
> I would like to keep it lightweight and hopefully some more people would adopt it.
> 
> I believe adding zabbix should be done as an optional dependency. We could simply try to import the Zabbix modules. If they are there, the process has support for Zabbix. If not, then not. That could simply be handled at runtime.
> 
> That way, we could have zabbix as an add-on in IPFire instead of adding it to the core distribution and if other people want to run the whole thing, they can enjoy a minimum amount of dependencies and don’t have to build/install any zabbix packages if they don’t need them.
> 
>> Both the python module and the commandline cli have the possibility to
>> get zabbix server connection info from the zabbix_agentd configfile so
>> config of the reporter would be something like:
>> [zabbix]
>> enabled = true
>> zabbix_agentd_config = /etc/zabbix_agentd/zabbix_agentd.conf
>> alert_item_key = ipfire.suricata.event.get
> 
> I mean, this is all up to you. I would definitely do this bit:
> 
> [zabbix]
> enabled = true
> 
> That way it is consistent with the remainder. Any other options, well I don’t know what you would need :) I do enjoy a daemon that does not need a lot of configuration though and therefore can be run with only a very small configuration as it relies on sensible defaults.
> 
>> Then the reporter can format the incoming suricata alert/event as json
>> and send it to the configured alert_item_key on the zabbix server as
>> configured in the zabbix_agentd.conf
> 
> We do store the entire JSON object as it comes from Suricata in the sqlite3 database. That way we have all the information that could possible be available to us even though we don’t need everything right now. The fast.log format definitely lacks a lot of important information, but it is nicely readable.
> 
> Depending on what processing Zabbix can do later and what information it permanently-ish stores, I would send the whole lot. The JSON objects are small enough and over the wire they would compress nicely, too. On the daemon side I wouldn’t implement this to be configurable because that only makes everything more complicated.
> 
>> Is this something you are open to? Then I can try to create a patch for
>> suricata-reporter. (where should I then submit it? Also on this list?)
> 
> Yes, very much so.
> 
> Patches can go on this list.
> 
> Let’s talk about any implementation details in advance. I feel a little bit precious about this program because it is nice, shiny and new and the code is very simple. I would like to keep it that way.
> 
>> If not I will have to continue working on the fast.log parsing.
> 
> Don’t. That will only make you unhappy.
> 
>> And while on the topic of monitoring suricata; I would like to get some
>> extra stats from it, which, for as far as I currently know, can be
>> retrieved using the suricata unix-socket that is currently disabled by
>> default on ipfire. Many seem to use a 'suricatasc' tool to query
>> suricata using that socket, but that tool is not available on ipfire.
> 
> What stats do you want/need?
> 
> I am happy to turn on the socket and built the tool. Does it come with the suricata package?

I can help by submitting a patch to enable the unix-socket in the build and uncomment the suricatasc package. This was added in with the 8.0.0 release but initially I just commented it out as we hadn't used it before. It could always be uncommented later on as it will now be.

I am doing a range of patch update submission work so I can add that easily enough into what I am doing.

Regards,

Adolf.

> 
> Please submit patches :) I think this could be done first before diving into Zabbix because it should be a very simple change.
> 
>> Is it possible to have it on ipfire?, or should I start experimenting
>> using socat?
> 
> No, if there are tools built already, use the code that is available.
> 
>> And if succesful, is it then allowed for a future zabbix_agentd addon
>> pak to enable that socket in the suricata config?
> 
> Enable the socket by default. It won’t hurt anyone.
> 
>> If you dislike the idea of enabling and querying the socket, another
>> possibility is having suricata dump stats in a seperate stats.log which
>> I should then be able to parse using Zabbix.
> 
> Hmm, but then we are back to parsing logs. That is never fun.
> 
>> Before I start any implementations, what are your thoughts about all
>> this ?
> 
> Very very happy with this here.
> 
> Let me know how I can help.
> 
> -Michael
> 
>> Regards
>> Robin
>>
>> -- 
>> Dit bericht is gescanned op virussen en andere gevaarlijke
>> inhoud door MailScanner en lijkt schoon te zijn.
>>
> 
> 

Re: Zabbix support for suricata-reporter

[Robin Roevens]

Hi Adolf

Adolf Belka schreef op za 04-10-2025 om 13:41 [+0200]:
> 
> I can help by submitting a patch to enable the unix-socket in the
> build and uncomment the suricatasc package. This was added in with
> the 8.0.0 release but initially I just commented it out as we hadn't
> used it before. It could always be uncommented later on as it will
> now be.
> 
> I am doing a range of patch update submission work so I can add that
> easily enough into what I am doing.
Perfect. Then please enable it. The tool also requires the unix-command
socket to be enabled in suricata config
(https://docs.suricata.io/en/latest/unix-socket.html#interacting-via-unix-socket)

Can you enable that too in the same go?

> 
> Regards,
> 
> Adolf.
> 
> 
> 

Regards
Robin

-- 
Dit bericht is gescanned op virussen en andere gevaarlijke
inhoud door MailScanner en lijkt schoon te zijn.

Re: Zabbix support for suricata-reporter

[Adolf Belka]

Hi Robin,

On 17/10/2025 00:06, Robin Roevens wrote:
> Hi Adolf
> 
> Adolf Belka schreef op za 04-10-2025 om 13:41 [+0200]:
>>
>> I can help by submitting a patch to enable the unix-socket in the
>> build and uncomment the suricatasc package. This was added in with
>> the 8.0.0 release but initially I just commented it out as we hadn't
>> used it before. It could always be uncommented later on as it will
>> now be.
>>
>> I am doing a range of patch update submission work so I can add that
>> easily enough into what I am doing.
> Perfect. Then please enable it. The tool also requires the unix-command
> socket to be enabled in suricata config
> (https://docs.suricata.io/en/latest/unix-socket.html#interacting-via-unix-socket)
> 
> Can you enable that too in the same go?

Yes.

Patch submitted.

Regards,

Adolf.

> 
>>
>> Regards,
>>
>> Adolf.
>>
>>
>>
> 
> Regards
> Robin
> 

[SPAM Warning!]Re: Zabbix support for suricata-reporter

[Robin Roevens]

Hi Michael

Michael Tremer schreef op za 04-10-2025 om 11:52 [+0100]:
> Hello Robin,
> 
> > On 2 Oct 2025, at 22:04, Robin Roevens <robin.roevens@disroot.org>
> > wrote:
> > 
> > Hi Michael
> > 
> > I saw there is a suricata-reporter in the upcoming CU. And I was
> > wondering if I could add an additional reporter into it for sending
> > alerts straight to Zabbix, next to syslog and email.
> 
> I think that is a great idea!
That makes at least 2 of us :-)

> 
> > I have already been experimenting with parsing fast.log using the
> > zabbix_agentd, which seems to work quite well. But since there is
> > now a
> > reporter, it would be nice to have it support sending alerts to
> > zabbix
> > directly instead of zabbix separately monitoring the fast.log file.
> 
> Parsing log files like that is never fun. I also think that I
> remember that suricata has deprecated support for writing fast.log,
> but then it does not seem to be deprecated any more. Looking at the
> alternative EVE format, fast.log is just there to have something that
> is somewhat human-readable, but at the same time does not carry a lot
> of information.
I had considered EVE logs but it was not really clear to me how to
distinguish normal events and actual alert events from than, and I saw
that it is currently disabled, but that the reporter will soon enable
it to connect to a socket. So I won't be able to listen in with the
zabbix_agent.. I think.. 
Anyway, no longer relevant if it is build in into the reporter.
 
> 
> > If that would be ok for you. There are 2 possible ways to do this:
> > - using the zabbix_utils python library:
> > https://blog.zabbix.com/python-zabbix-utils/27056/
> > - or using the zabbix_sender command utility that currently gets
> > installed when installing zabbix_agentd
> 
> The IPS could generate a lot of alerts. Especially when there is
> bursts in (malicious) traffic and I am already unhappy with calling
> sendmail - but there I have no other choice.
> 
> The daemon initially had some worker daemons so that reading from
> Suricata would be entirely decoupled from processing any messages.
> However, Python’s multiprocessing capabilities are really bad and so
> now it is a threaded implementation. Python isn’t good at that
> either, but when we are calling a subprocess the GIL is being
> released and so we should not actually spend too much time in the
> Python code itself.
> 
> This has to be considered when adding yet another output so that the
> process does not get stalled when Zabbix does not respond, is
> overwhelmed or anything else. We could simply combat that by adding a
> couple more worker threads if necessary - but I am getting ahead of
> myself.
> 
> So the Python module would be by far preferable. It seems to have a
> nice API, but we will have to check how it works underneath to reach
> the objectives outlined above.
By default the zabbix sender functionality of the library has a timeout
of 10s. When the Zabbix server is healthy, that timeout is never
reached. But when unavailable or overloaded, that can happen. This
timeout is configurable.
It also has the possibility to bulk-send entries, which it will
internally split into chunks of max 250 items which will then each be
sent separately. Each chunk then has the 10s timout. Chunksize can be
configured also.
So to minimize zabbix communications, it could be a possibility to
cache events within a timeframe (say 30s) and only once per 30s send
all events in the queue. 30s is no longer 'real-time', so I would like
to avoid that, but it may be a good idea to at least buffer within 1s
to prevent possible hammering the server every millisecond in some
extreme cases?

There is also an async version of the sender functionality included in
the library. I've not yet done any async python code, so I'm not sure I
understand it correctly, but I assume it can make the actual sending a
sort of background job, while the reporter can continue its other work?

> 
> > I assume, using the python library will probably be the most
> > performant
> > option; But then I should also create a zabbix_utils python library
> > pak-file?
> 
> Hmm, this is where it is getting interesting.
> 
> The suricata-reporter stuff is supposed to be rather distro agnostic.
> Besides Python itself, it only has a dependency to sqlite3 (which
> normally comes with the Python standard library) and reportlab to
> generate the PDFs. The goal is to run it on IPFire 2 for now, but it
> would also be pretty much ready as is to run in IPFire 3 some day. Of
> course it runs on any other distro - although I don’t know what it
> would do on a non-Linux OS.
> 
> I would like to keep it lightweight and hopefully some more people
> would adopt it.
> 
> I believe adding zabbix should be done as an optional dependency. We
> could simply try to import the Zabbix modules. If they are there, the
> process has support for Zabbix. If not, then not. That could simply
> be handled at runtime.
> 
> That way, we could have zabbix as an add-on in IPFire instead of
> adding it to the core distribution and if other people want to run
> the whole thing, they can enjoy a minimum amount of dependencies and
> don’t have to build/install any zabbix packages if they don’t need
> them.

I agree that suricata-reporter in itself should be as lightweight and
distro agnostic as possible. Adding zabbix as an optional dependency is
an option. But I was wondering if it wouldn't be more elegant to have
suricata-reporter support some sort of reporter modules or plugins,
which could then contain the email, pdf and zabbix functionalities.
That way, you could ship suricata-reporter without zabbix
functionalities, and I could create a suricata-reporter-zabbix-plugin
with the zabbix specific code in it that, when installed, is picked up
by suricata-reporter. For IPFire I could then just include the reporter
plugin in the zabbix_agentd pak on in a dependent separate pak which in
turn depends on a new zabbix-utils pak containing the zabbix python
library, and for other distro's, it can be downloaded from my git,
and/or I add it to an ipfire git repo?

> 
> > Both the python module and the commandline cli have the possibility
> > to
> > get zabbix server connection info from the zabbix_agentd configfile
> > so
> > config of the reporter would be something like:
> > [zabbix]
> > enabled = true
> > zabbix_agentd_config = /etc/zabbix_agentd/zabbix_agentd.conf
> > alert_item_key = ipfire.suricata.event.get
> 
> I mean, this is all up to you. I would definitely do this bit:
> 
> [zabbix]
> enabled = true
> 
> That way it is consistent with the remainder. Any other options, well
> I don’t know what you would need :) I do enjoy a daemon that does not
> need a lot of configuration though and therefore can be run with only
> a very small configuration as it relies on sensible defaults.
I think what I gave as an example, could suffice. It may be usefull to
support also explicit zabbix server settings instead of referencing the
zabbix_agentd config. On IPFire, using the zabbix_agentd config is
probably the best option to avoid duplicate configurations. On other
distro's or use cases of the reporter, it may well be that it is used
without a zabbix_agent installed, hence without a pre-existing config
that can be used to get zabbix server details from.

And in case of the plugin-architecture, I think easiest is to also
support a loadable config-directory, so config files from other pak-
files can be dropped in it?

> 
> > Then the reporter can format the incoming suricata alert/event as
> > json
> > and send it to the configured alert_item_key on the zabbix server
> > as
> > configured in the zabbix_agentd.conf
> 
> We do store the entire JSON object as it comes from Suricata in the
> sqlite3 database. That way we have all the information that could
> possible be available to us even though we don’t need everything
> right now. The fast.log format definitely lacks a lot of important
> information, but it is nicely readable.
> 
> Depending on what processing Zabbix can do later and what information
> it permanently-ish stores, I would send the whole lot. The JSON
> objects are small enough and over the wire they would compress
> nicely, too. On the daemon side I wouldn’t implement this to be
> configurable because that only makes everything more complicated.

I will have to study what would exactly come through. But zabbix has a
quite powerfull preprocessing engine which can format, validate, split,
and can do almost anything with the incoming data. So that would
probably indeed be the easiest method

> 
> > Is this something you are open to? Then I can try to create a patch
> > for
> > suricata-reporter. (where should I then submit it? Also on this
> > list?)
> 
> Yes, very much so.
> 
> Patches can go on this list.
> 
> Let’s talk about any implementation details in advance. I feel a
> little bit precious about this program because it is nice, shiny and
> new and the code is very simple. I would like to keep it that way.
And here I go suggesting wild extensible plugin frameworks :-)
However, I don't think that should be very complicated. And it would
create an ability to keep the reporter clean, and leave possible future
extra plugins in the hands of others.
 
> 
> > If not I will have to continue working on the fast.log parsing.
> 
> Don’t. That will only make you unhappy.
Thanks.. I need some happiness once in a while :-)
> 
> > And while on the topic of monitoring suricata; I would like to get
> > some
> > extra stats from it, which, for as far as I currently know, can be
> > retrieved using the suricata unix-socket that is currently disabled
> > by
> > default on ipfire. Many seem to use a 'suricatasc' tool to query
> > suricata using that socket, but that tool is not available on
> > ipfire.
> 
> What stats do you want/need?
For starters I was thinking iface-list and iface-stat to get interface-
specific stats about number of packets scanned, dropped, bypassed. But
I will have to experiment a bit with it to see what stats can be
valuable.
> 
> I am happy to turn on the socket and built the tool. Does it come
> with the suricata package?
I think it is a build option. I saw Adolf already proposed to enable
it.

> 
> Please submit patches :) I think this could be done first before
> diving into Zabbix because it should be a very simple change.
> 
> > Is it possible to have it on ipfire?, or should I start
> > experimenting
> > using socat?
> 
> No, if there are tools built already, use the code that is available.
ok

> 
> > And if succesful, is it then allowed for a future zabbix_agentd
> > addon
> > pak to enable that socket in the suricata config?
> 
> Enable the socket by default. It won’t hurt anyone.
ok.. most certainly it is better to have it enabled by default when the
suricatasc tool is also available on IPFire. Giving users also the
possibility to use it manually or to query suricatta using other tools
than Zabbix.

> 
> > If you dislike the idea of enabling and querying the socket,
> > another
> > possibility is having suricata dump stats in a seperate stats.log
> > which
> > I should then be able to parse using Zabbix.
> 
> Hmm, but then we are back to parsing logs. That is never fun.
agreed
> 
> > Before I start any implementations, what are your thoughts about
> > all
> > this ?
> 
> Very very happy with this here.
> 
> Let me know how I can help.
> 
> -Michael
> 
> > Regards
> > Robin
> > 
> > -- 
> > Dit bericht is gescanned op virussen en andere gevaarlijke
> > inhoud door MailScanner en lijkt schoon te zijn.
> > 
> 
Regards
Robin

-- 
Dit bericht is gescanned op virussen en andere gevaarlijke
inhoud door MailScanner en lijkt schoon te zijn.

Re: [SPAM Warning!]Re: Zabbix support for suricata-reporter

[Michael Tremer]

Hello Robin,

> On 16 Oct 2025, at 22:59, Robin Roevens <robin.roevens@disroot.org> wrote:
> 
> Hi Michael
> 
> Michael Tremer schreef op za 04-10-2025 om 11:52 [+0100]:
>> Hello Robin,
>> 
>>> On 2 Oct 2025, at 22:04, Robin Roevens <robin.roevens@disroot.org>
>>> wrote:
>>> 
>>> Hi Michael
>>> 
>>> I saw there is a suricata-reporter in the upcoming CU. And I was
>>> wondering if I could add an additional reporter into it for sending
>>> alerts straight to Zabbix, next to syslog and email.
>> 
>> I think that is a great idea!
> That makes at least 2 of us :-)

I am sure there are more. On Reddit, your Zabbix add-on seems to be very popular, too:

  https://www.reddit.com/r/ipfire/comments/1oapu1t/ipfire_and_zabbix_integration/

>> 
>>> I have already been experimenting with parsing fast.log using the
>>> zabbix_agentd, which seems to work quite well. But since there is
>>> now a
>>> reporter, it would be nice to have it support sending alerts to
>>> zabbix
>>> directly instead of zabbix separately monitoring the fast.log file.
>> 
>> Parsing log files like that is never fun. I also think that I
>> remember that suricata has deprecated support for writing fast.log,
>> but then it does not seem to be deprecated any more. Looking at the
>> alternative EVE format, fast.log is just there to have something that
>> is somewhat human-readable, but at the same time does not carry a lot
>> of information.
> I had considered EVE logs but it was not really clear to me how to
> distinguish normal events and actual alert events from than, and I saw
> that it is currently disabled, but that the reporter will soon enable
> it to connect to a socket. So I won't be able to listen in with the
> zabbix_agent.. I think.. 
> Anyway, no longer relevant if it is build in into the reporter.

Yes, this is kind of abstracted away for you. The problem with parsing a log file is also where to resume after a restart of the service? What happens when the log file is being rotated?

Therefore I have chosen to write daemon that suricata is directly writing to. This comes with some other challenges, but I thought that it would be much easier to solve these instead.

We are receiving the full EVE log and we can process any kind of events that we are interested in. Currently, this is only the alerts.

>> 
>>> If that would be ok for you. There are 2 possible ways to do this:
>>> - using the zabbix_utils python library:
>>> https://blog.zabbix.com/python-zabbix-utils/27056/
>>> - or using the zabbix_sender command utility that currently gets
>>> installed when installing zabbix_agentd
>> 
>> The IPS could generate a lot of alerts. Especially when there is
>> bursts in (malicious) traffic and I am already unhappy with calling
>> sendmail - but there I have no other choice.
>> 
>> The daemon initially had some worker daemons so that reading from
>> Suricata would be entirely decoupled from processing any messages.
>> However, Python’s multiprocessing capabilities are really bad and so
>> now it is a threaded implementation. Python isn’t good at that
>> either, but when we are calling a subprocess the GIL is being
>> released and so we should not actually spend too much time in the
>> Python code itself.
>> 
>> This has to be considered when adding yet another output so that the
>> process does not get stalled when Zabbix does not respond, is
>> overwhelmed or anything else. We could simply combat that by adding a
>> couple more worker threads if necessary - but I am getting ahead of
>> myself.
>> 
>> So the Python module would be by far preferable. It seems to have a
>> nice API, but we will have to check how it works underneath to reach
>> the objectives outlined above.
> By default the zabbix sender functionality of the library has a timeout
> of 10s. When the Zabbix server is healthy, that timeout is never
> reached. But when unavailable or overloaded, that can happen. This
> timeout is configurable.

We would have to factor this in though. It must be possible for Zabbix to be slow because it either is busy, or simply not easily reachable over the network. A classic case could be where the firewall is under some (D)DoS attack and we still don’t want to simply drop any kind of alert because frankly, alerting the admin is exactly what this service is for.

> It also has the possibility to bulk-send entries, which it will
> internally split into chunks of max 250 items which will then each be
> sent separately. Each chunk then has the 10s timout. Chunksize can be
> configured also.

Sending entries in bulk would be great to reduce overhead. I don’t know about Zabbix in particular, but often client libraries like this use Python requests which will open up a new TCP/TLS connection per request. That has a huge amount of overhead when you have hundreds of alerts.

If we would create some queue where everything that is interesting is being called and try to once a second flush everything that is in the queue to Zabbix, we should be able to batch a lot of alerts and (unless we exceed the limit of 250 alerts per second) send them over one single TCP connection per second. That would be very efficient even with large numbers of alerts.

Since this software is currently very young and since I have not seen it being deployed in really large environments, I don’t quite know what the target should be. I would consider it a possibility of some people having a sustained rate of thousands of alerts per second. Hopefully nobody will be trying to send this via email and more, but the SQLite database can easily ingest this and therefore we can report on it well.

> So to minimize zabbix communications, it could be a possibility to
> cache events within a timeframe (say 30s) and only once per 30s send
> all events in the queue. 30s is no longer 'real-time', so I would like
> to avoid that, but it may be a good idea to at least buffer within 1s
> to prevent possible hammering the server every millisecond in some
> extreme cases?

Yes, absolutely.

And although I don’t like threads in Python, I suppose this should be implemented in a separate thread. If we build this well, when the Zabbix client is waiting, it won’t block the GIL at all.

> There is also an async version of the sender functionality included in
> the library. I've not yet done any async python code, so I'm not sure I
> understand it correctly, but I assume it can make the actual sending a
> sort of background job, while the reporter can continue its other work?

That would be an option, too. I like to write asynchronous Python, but it makes things easily a lot more complex. Therefore, the reporter has an async main thread and forks some worker threads which are implemented synchronously. We could easily change the latter to be fully async. It wouldn’t hurt me. I just didn’t have the need to do it.

This would however allow us to dispatch multiple sets of alerts at the same time and we wouldn’t have to wait for one request to be completed. This could then have the potential to send a lot of data really fast to the Zabbix server and I am not sure how that would cope. Do we have some idea how long sending a single alert would take, a hundred alerts?

>>> I assume, using the python library will probably be the most
>>> performant
>>> option; But then I should also create a zabbix_utils python library
>>> pak-file?
>> 
>> Hmm, this is where it is getting interesting.
>> 
>> The suricata-reporter stuff is supposed to be rather distro agnostic.
>> Besides Python itself, it only has a dependency to sqlite3 (which
>> normally comes with the Python standard library) and reportlab to
>> generate the PDFs. The goal is to run it on IPFire 2 for now, but it
>> would also be pretty much ready as is to run in IPFire 3 some day. Of
>> course it runs on any other distro - although I don’t know what it
>> would do on a non-Linux OS.
>> 
>> I would like to keep it lightweight and hopefully some more people
>> would adopt it.
>> 
>> I believe adding zabbix should be done as an optional dependency. We
>> could simply try to import the Zabbix modules. If they are there, the
>> process has support for Zabbix. If not, then not. That could simply
>> be handled at runtime.
>> 
>> That way, we could have zabbix as an add-on in IPFire instead of
>> adding it to the core distribution and if other people want to run
>> the whole thing, they can enjoy a minimum amount of dependencies and
>> don’t have to build/install any zabbix packages if they don’t need
>> them.
> 
> I agree that suricata-reporter in itself should be as lightweight and
> distro agnostic as possible. Adding zabbix as an optional dependency is
> an option. But I was wondering if it wouldn't be more elegant to have
> suricata-reporter support some sort of reporter modules or plugins,
> which could then contain the email, pdf and zabbix functionalities.

Hmm, it kind of is modular. There is a function that is being called whenever an alert has been received and depending on configuration, this function calls the functions to send emails, syslog, etc.

  https://git.ipfire.org/?p=suricata-reporter.git;a=blob;f=src/suricata-reporter.in;h=e1c25b48b49f7c6818aadfc8ad80439f5c35230b;hb=HEAD#l427

We could in theory create something here where modules register a function which will be called. Not a problem.

For simplicity, and to have some quick result in the first place, I have chosen this approach. See where the challenges are and then make the code more expandable.

> That way, you could ship suricata-reporter without zabbix
> functionalities, and I could create a suricata-reporter-zabbix-plugin
> with the zabbix specific code in it that, when installed, is picked up
> by suricata-reporter. For IPFire I could then just include the reporter
> plugin in the zabbix_agentd pak on in a dependent separate pak which in
> turn depends on a new zabbix-utils pak containing the zabbix python
> library, and for other distro's, it can be downloaded from my git,
> and/or I add it to an ipfire git repo?

This is where it will probably get messy. As mentioned before, the performance of this tool lies in not blocking the process. If we would have externally maintained modules, we would not have control over this and people might add some stuff that harms the performance of the software. Arguably that won’t be our problem then, but why make people’s lives hard?

I therefore would prefer to keep all the code in one repository so that we have control over it. It is all still very young and we want to have the option to make changes (even big ones) if we need to. For example make the entire software asynchronous instead of having synchronous worker threads. External modules would have to keep up with those changes, and since open source development has very sadly for me turned into building my own things on my own, and other people doing the same with usually none (and if they really have to minimal) conversation, we would probably just keep breaking things or feeling forced to keep everything as is.

I would say that we should integrate Zabbix into the main project and if there are is really a large number of other people coming to add more plugins, we can still add external modules. That way, we would at least have matured the software a little bit and be able to offer a more stable API.

Right now, nobody but us is using this software and so I don’t want to accommodate needs of people I don’t even know yet. What could they possibly need? I don’t know. Building out every possible option under the sun for zero users is not a good investment of anyone’s time.

>> 
>>> Both the python module and the commandline cli have the possibility
>>> to
>>> get zabbix server connection info from the zabbix_agentd configfile
>>> so
>>> config of the reporter would be something like:
>>> [zabbix]
>>> enabled = true
>>> zabbix_agentd_config = /etc/zabbix_agentd/zabbix_agentd.conf
>>> alert_item_key = ipfire.suricata.event.get
>> 
>> I mean, this is all up to you. I would definitely do this bit:
>> 
>> [zabbix]
>> enabled = true
>> 
>> That way it is consistent with the remainder. Any other options, well
>> I don’t know what you would need :) I do enjoy a daemon that does not
>> need a lot of configuration though and therefore can be run with only
>> a very small configuration as it relies on sensible defaults.
> I think what I gave as an example, could suffice. It may be usefull to
> support also explicit zabbix server settings instead of referencing the
> zabbix_agentd config. On IPFire, using the zabbix_agentd config is
> probably the best option to avoid duplicate configurations. On other
> distro's or use cases of the reporter, it may well be that it is used
> without a zabbix_agent installed, hence without a pre-existing config
> that can be used to get zabbix server details from.

I am flexible here. We are using Python’s configparser and that could either take a filename (or have a default), or simple take server, username, password or any other kind of credential to contact the service.

> And in case of the plugin-architecture, I think easiest is to also
> support a loadable config-directory, so config files from other pak-
> files can be dropped in it?

We can do this anyways. Could you please create a bug report for this so that we track this? It is not too urgent to implement, but whenever I have time, I am happy to do it.

  https://bugzilla.ipfire.org/enter_bug.cgi?product=Suricata%20Reporter

>> 
>>> Then the reporter can format the incoming suricata alert/event as
>>> json
>>> and send it to the configured alert_item_key on the zabbix server
>>> as
>>> configured in the zabbix_agentd.conf
>> 
>> We do store the entire JSON object as it comes from Suricata in the
>> sqlite3 database. That way we have all the information that could
>> possible be available to us even though we don’t need everything
>> right now. The fast.log format definitely lacks a lot of important
>> information, but it is nicely readable.
>> 
>> Depending on what processing Zabbix can do later and what information
>> it permanently-ish stores, I would send the whole lot. The JSON
>> objects are small enough and over the wire they would compress
>> nicely, too. On the daemon side I wouldn’t implement this to be
>> configurable because that only makes everything more complicated.
> 
> I will have to study what would exactly come through. But zabbix has a
> quite powerfull preprocessing engine which can format, validate, split,
> and can do almost anything with the incoming data. So that would
> probably indeed be the easiest method

We could of course do some processing in the reporter, or send the raw stuff. Up to you. It could even be configurable.

But before we figure that out, would you like to package the Zabbix Python modules that we need for IPFire, or are they already part of the zabbix agent package?

>> 
>>> Is this something you are open to? Then I can try to create a patch
>>> for
>>> suricata-reporter. (where should I then submit it? Also on this
>>> list?)
>> 
>> Yes, very much so.
>> 
>> Patches can go on this list.
>> 
>> Let’s talk about any implementation details in advance. I feel a
>> little bit precious about this program because it is nice, shiny and
>> new and the code is very simple. I would like to keep it that way.
> And here I go suggesting wild extensible plugin frameworks :-)
> However, I don't think that should be very complicated. And it would
> create an ability to keep the reporter clean, and leave possible future
> extra plugins in the hands of others.

It could. If those “others” turn up, let’s build it. If not, we won’t need it.

>> 
>>> If not I will have to continue working on the fast.log parsing.
>> 
>> Don’t. That will only make you unhappy.
> Thanks.. I need some happiness once in a while :-)
>> 
>>> And while on the topic of monitoring suricata; I would like to get
>>> some
>>> extra stats from it, which, for as far as I currently know, can be
>>> retrieved using the suricata unix-socket that is currently disabled
>>> by
>>> default on ipfire. Many seem to use a 'suricatasc' tool to query
>>> suricata using that socket, but that tool is not available on
>>> ipfire.
>> 
>> What stats do you want/need?
> For starters I was thinking iface-list and iface-stat to get interface-
> specific stats about number of packets scanned, dropped, bypassed. But
> I will have to experiment a bit with it to see what stats can be
> valuable.

This is a bit of a side note, but it has tickled my curiosity as I am currently working on replacing collectd. I want something that is more flexible, more custom to us and allows us to collect a lot more “interesting” data. Whatever that would be.

Out of “fun” I wrote a little suricata plugin (yes, it has a little plugin thing):

  https://git.ipfire.org/?p=collecty.git;a=blob;f=src/daemon/sources/suricata.c;h=e317566a5d0ad0fa36ef97542e145ee68314176b;hb=HEAD

It calls suricatasc --command=dump-counters (asynchronously in C) and parses the returned JSON. But then I didn’t quite know what I should actually store?!

I would be happy to hear your ideas. But this is probably another conversation.

>> 
>> I am happy to turn on the socket and built the tool. Does it come
>> with the suricata package?
> I think it is a build option. I saw Adolf already proposed to enable
> it.

Yes, it is enabled and I have tested it. It returns a whole tapestry with metrics.

> 
>> 
>> Please submit patches :) I think this could be done first before
>> diving into Zabbix because it should be a very simple change.
>> 
>>> Is it possible to have it on ipfire?, or should I start
>>> experimenting
>>> using socat?
>> 
>> No, if there are tools built already, use the code that is available.
> ok
> 
>> 
>>> And if succesful, is it then allowed for a future zabbix_agentd
>>> addon
>>> pak to enable that socket in the suricata config?
>> 
>> Enable the socket by default. It won’t hurt anyone.
> ok.. most certainly it is better to have it enabled by default when the
> suricatasc tool is also available on IPFire. Giving users also the
> possibility to use it manually or to query suricatta using other tools
> than Zabbix.

Yes, this does not cost anything extra. It is done.

>> 
>>> If you dislike the idea of enabling and querying the socket,
>>> another
>>> possibility is having suricata dump stats in a seperate stats.log
>>> which
>>> I should then be able to parse using Zabbix.
>> 
>> Hmm, but then we are back to parsing logs. That is never fun.
> agreed
>> 
>>> Before I start any implementations, what are your thoughts about
>>> all
>>> this ?
>> 
>> Very very happy with this here.
>> 
>> Let me know how I can help.
>> 
>> -Michael
>> 
>>> Regards
>>> Robin
>>> 
>>> -- 
>>> Dit bericht is gescanned op virussen en andere gevaarlijke
>>> inhoud door MailScanner en lijkt schoon te zijn.
>>> 
>> 
> Regards
> Robin
> 
> -- 
> Dit bericht is gescanned op virussen en andere gevaarlijke
> inhoud door MailScanner en lijkt schoon te zijn.
>