Re: [PATCH 0/5] Add Zabbix functionality to suricata-reporter

[Michael Tremer <michael.tremer@ipfire.org>]

Hello Robin,

Thank you very much for sending these patches.

Before we dig into the code, I have a couple of questions about the design...

> On 30 Jul 2026, at 20:15, Robin Roevens <robin.roevens@disroot.org> wrote:
> 
> Hi all,
> 
> As discussed here earlier, I've worked on implementing sending
> Suricata alerts straight to Zabbix from within suricata-reporter instead
> of trying to parse the suricata logging separately using the Zabbix agent.
> 
> For this I use the zabbix-utils python library, which I submited here
> also as a separate pak (but meanwhile already requires an update, which
> I will post soon). This set of patches makes suricata-reporter able to
> directly communicate to a Zabbix server without having the zabbix_agentd
> pak installed, sending suricata alerts in real-time.

Yes, this is a good choice and I like that suricate-reporter will try to load support for Zabbix and if the module is not available, it simply disables support for Zabbix. That allows us to have a smaller configuration file if things like this are auto-detected.

> As Zabbix supports sending items in bulk, I have opted to create an
> async background task that will send all events from last 1 second in
> bulk so that even in the case that there are hundreds of incoming
> alerts, Zabbix server is only contacted once per second.

Okay, this makes sense. But I believe that there is already a small race in the implementation:

If the client side (in this case suricata-reporter) does not finish the call of flush_pending_to_zabbix() within that second, it will be called again which will result in the same rows being selected again, transmitted again, and assuming that there are just thousands of alarms it will take over a second again, the function will be called again, and so on. So the application will stall very quickly.

Although we should not see thousands of alerts per second under normal conditions, there could be other reasons why this is taking some time. For example, the Zabbix host could be in a different location and round-trips around half the planet are taking some time; it could be busy writing other things to its database or the database has just decided to do a little cleanup job. One second isn’t a lot of time then and we will have to make the system a little bit more resilient against this.

> When for some reason sending to Zabbix server fails, it will be retried
> 3 times and then the background task will be suspended until a new
> suricata event comes in. That will wake the task again and retry to send all
> pending events. In environments with many events, that may actually not
> have that much of an effect. But in the average environment, this will
> give the Zabbix Server some breathing space as it failing to receive our
> events, may indicate a Zabbix server overload.

Good thinking here.

> For this I have to keep track which events are sent and which are
> pending. So I added a column in the database that keeps track of that.

So, this is a very crucial thing we probably need to discuss :)

What is the rationale behind this? Obviously there are some easy answers:

1) We don’t want to loose any history if the network or Zabbix is down

2) We can even restart the reporter without losing any alerts

But then I am already running out of ideas why this could be a good idea. The cons that I can see are:

* A lot of additional I/O on the database. Although we would be updating rows very briefly after they have been written to the database, it will create a copy of the row and change the append-only architecture of the database. It will have a lot more cleaning up to do to evict all updated rows.

* You will only ever go back by about 1h by default. Could we just not keep things in RAM for that long?

I am not saying that I hate the idea, but I am not sure whether it is worth paying the price. The good side is that if people are not using Zabbix, there is no overhead except the space for the extra column. But if we would add another monitoring solution, we would potentially have to add another field, and another, and another?

So a possible other solution that I can come up with would be: Creating a separate table with all pending events that have to be transmitted. And every once in a while we truncate it should it become too long. We could even keep a list of IDs in memory only if we want to go down that route.

> I have also added an alert_max_age config parameter that allows the user
> to set how long suricata-reporter should retry to send events to Zabbix.
> Events older than that set age, will no longer be sent to Zabbix.
> This also give the user the implicit option to send older events when
> only just enabling the zabbix sending functionality, since the DB column
> exists and no event was ever sent to Zabbix, all events will be
> 'pending". At first run with zabbix functionality enabled, all events up
> to alert_max_age that are in the database will be sent to zabbix
> immediatly.

I like the mechanism, but whenever I am building something like this, I am never sure what would be a reasonable window.

Locally, suricate-reporter is keeping the events for pretty much forever. So we could even go back three days or something. Or we could give up really quickly. After maybe a minute. I never know what is right, but for the implementation, the length of the window plays a role - see above.

With email and syslog we do more of a “fire and forget” approach. If we send the syslog message and syslog wasn’t ready to receive it, we wouldn’t know and we would not try again...

> All events sent to Zabbix contain the timestamp of retrieval by
> suricata-reporter, so Zabbix will register and order them as received on that
> timestamp independently of the actual time Zabbix itself received the
> event.
> 
> This is my first adventure in Python async programming, so I hope I did
> not make any flagrant mistakes. But the code has been running here for
> weeks now without any problem. I have not actually tested large bursts
> of events, as I could not simulate that.. But I did make Zabbix server
> slow, unavailable and finally replaced it with netcat (to accept the connection, but
> not react on it) and I had the connection with the server off for a few
> hours to then re-establish the connection to see hundereds of pending events 
> being registered in only a few milliseconds.
> I did not notice any problems with suricata-reporter in any of these
> cases.

This is good testing. Usually, if I need to create a lot of events, I enable the “PING” rule in “icmp_info” and just send a lot of ping packets to the firewall. You could try a flood ping with “ping -f”.

I will send some more comments about the code in the other emails.

Best,
-Michael

> 
> Regards
> 
> Robin
> 
> -- 
> Dit bericht is gescanned op virussen en andere gevaarlijke
> inhoud door MailScanner en lijkt schoon te zijn.
> 
>