From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.haj.ipfire.org (localhost [IPv6:::1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4cnhhB4gmWz30FL for ; Thu, 16 Oct 2025 21:59:54 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [IPv6:2001:678:b28::25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (secp384r1) server-digest SHA384 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail01.haj.ipfire.org", Issuer "R13" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4cnhh72BCyz2xRj for ; Thu, 16 Oct 2025 21:59:51 +0000 (UTC) Received: from layka.disroot.org (layka.disroot.org [178.21.23.139]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mail01.ipfire.org (Postfix) with ESMTPS id 4cnhh40sLSz3R3 for ; Thu, 16 Oct 2025 21:59:48 +0000 (UTC) Authentication-Results: mail01.ipfire.org; dkim=fail ("headers rsa verify failed") header.d=disroot.org header.s=mail header.b=B4aCGidE; dmarc=pass (policy=reject) header.from=disroot.org; spf=pass (mail01.ipfire.org: domain of robin.roevens@disroot.org designates 178.21.23.139 as permitted sender) smtp.mailfrom=robin.roevens@disroot.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=lists.ipfire.org; s=202003rsa; t=1760651988; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V10cKIxSHzqq7bmc4i9eJQ1X90Nm5cCehlmDZE1Ji44=; b=AFURExL4FkTIOKWugVaLYWA0AsW+PjUfL9zHmOYYOiVXFTG65YhrYjb9YMpplfuOgzTvbn zemv9PjztCWxnkEZIM/SooJg9326J//NJYQoRN0sqoMGKn/Im+ohyXh0h5XD34gwmhi3pj QAHDfru65KkwpSebN6gliliPxn7VCbZwrMgUrPKwXXpT3r8CxXjKGSmXHzRj1rR0Dxrl4o 7SpiJ1JBxQ108wh8/hCmeCMO2iBkHFExi8sXd3XVivdtysZarxUy76rhuYMHxIQ66rKE+8 LyJZiwIwL8Kn/XOjSDKzoXHa11oBNLndQlnqNJ6ynzy4UZmeo90CnUhy4W98Fw== ARC-Authentication-Results: i=1; mail01.ipfire.org; dkim=fail ("headers rsa verify failed") header.d=disroot.org header.s=mail header.b=B4aCGidE; dmarc=pass (policy=reject) header.from=disroot.org; spf=pass (mail01.ipfire.org: domain of robin.roevens@disroot.org designates 178.21.23.139 as permitted sender) smtp.mailfrom=robin.roevens@disroot.org ARC-Seal: i=1; s=202003rsa; d=lists.ipfire.org; t=1760651988; a=rsa-sha256; cv=none; b=VO7wIdG0b0WsvrFKR7DKM1bRYFvWGqPAHKUvHykHNWnuSCYxqpu/jA5SUFUSGsdiTXHS2z mkh3SAyJenShvXhlRamiR3JZbdxbymkn3I4iSfoGxzYLQgKInclXwZjRlcvwxrBJ5p2Mvu MwQOlcpVb/N0w5cOWcsXFRnMrDz+If+CQ5ivIJ7qdxg39vWmI7rLCOiNUMs4iQZMc5YH2e zQiwYLWfYIs3b9DjjIfI+ExzKgRGvKHC+fnpZQ9vwpa5xgcyKyo8r8nf9Klxw/qt9QbLRR imy+zroXqyTrVMxEEEICSUeUOvFIVcNSr8EU115L/NR9d7qCLOYmGn6uA7CX8Q== Received: from mail01.disroot.lan (localhost [127.0.0.1]) by disroot.org (Postfix) with ESMTP id 3F888263CD; Thu, 16 Oct 2025 23:59:47 +0200 (CEST) X-Virus-Scanned: SPAM Filter at disroot.org X-Spam-Flag: YES X-Spam-Score: 9.9 X-Spam-Level: ********* X-Spam-Status: Yes, score=9.9 tagged_above=5 required=6.5 tests=[ALL_TRUSTED=-1, BAYES_00=-1.9, DISROOT_FAKE_FROM=1, DISROOT_KEYWORDS=1, DISROOT_PHISH=10, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MAIN_FROM=1] autolearn=no autolearn_force=no Received: from layka.disroot.org ([127.0.0.1]) by localhost (disroot.org [127.0.0.1]) (amavis, port 10024) with ESMTP id cBPfcuep8rdz; Thu, 16 Oct 2025 23:59:46 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=disroot.org; s=mail; t=1760651986; bh=V10cKIxSHzqq7bmc4i9eJQ1X90Nm5cCehlmDZE1Ji44=; h=Subject:From:To:Cc:Date:In-Reply-To:References; b=B4aCGidExldenJnR7X832qVsc/2teK4/zU6pqVzwjD+OMxzNH/YzR8u+SRe/1x1YM +Z9Wz/I/LmwTUcpOaCCWdqHD3QKsI+GVWSPfBGWppq4HnmAiCAyza9WjBJIRS6Ov4s QkJeP9duyqeMOi4GMIkHqEGZP5ZLDebryr73YzNgeF2T52ZIfYdjM/X9qUUXmGzeS1 /xogYU5+W0iE4fJUmHjiZhFcPrkJuwMYE+4UPxd46qViY8SDE1y/G23P1nESgnM6+r 2KUJ5YnRi3y/v10/9lWTeSc5SFIfJOvFz9VPfQBLXyaMS4al7JKKrAp2z+MA/mF4hA GL2kFIokMqZ7w== Received: from chojin.roevenslambrechts.be (Chojin.roevenslambrechts.be [192.168.0.50]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (no client certificate requested) (Authenticated sender) by hachiman (MailScanner Milter) with SMTP id 74F0240CF8C; Thu, 16 Oct 2025 23:59:38 +0200 (CEST) Message-ID: Subject: [SPAM Warning!]Re: Zabbix support for suricata-reporter From: Robin Roevens To: Michael Tremer Cc: "IPFire: Development-List" Date: Thu, 16 Oct 2025 23:59:38 +0200 In-Reply-To: <708978AA-034A-4533-BF40-257A843AFF8E@ipfire.org> References: <56ad0ff847bb0b5b5808084321ac52efd63cd16f.camel@disroot.org> <708978AA-034A-4533-BF40-257A843AFF8E@ipfire.org> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.56.2 Precedence: list List-Id: List-Subscribe: , List-Unsubscribe: , List-Post: List-Help: Sender: Mail-Followup-To: MIME-Version: 1.0 X-RoevensLambrechts-MailScanner-ID: 74F0240CF8C.A7644 X-RoevensLambrechts-MailScanner: Found to be clean X-RoevensLambrechts-MailScanner-From: robin.roevens@disroot.org X-RoevensLambrechts-MailScanner-Watermark: 1761256780.75654@sycmRaTfN6duilGSnpcowA X-Rspamd-Queue-Id: 4cnhh40sLSz3R3 X-Spamd-Result: default: False [4.00 / 11.00]; SPAM_FLAG(5.00)[]; BAYES_HAM(-3.00)[99.99%]; SPF_REPUTATION_SPAM(1.67)[0.5551566162987]; NEURAL_SPAM(1.15)[0.577]; DMARC_POLICY_ALLOW_WITH_FAILURES(-0.50)[]; R_SPF_ALLOW(-0.20)[+a]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; IP_REPUTATION_HAM(-0.01)[asn: 50673(0.00), country: NL(-0.01), ip: 178.21.23.139(0.00)]; RCVD_COUNT_THREE(0.00)[3]; SUBJECT_HAS_EXCLAIM(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; FUZZY_RATELIMITED(0.00)[rspamd.com]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; R_DKIM_REJECT(0.00)[disroot.org:s=mail]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_REPUTATION(0.00)[0]; DKIM_TRACE(0.00)[disroot.org:-]; ASN(0.00)[asn:50673, ipnet:178.21.23.0/24, country:NL]; DMARC_POLICY_ALLOW(0.00)[disroot.org,reject]; ARC_SIGNED(0.00)[lists.ipfire.org:s=202003rsa:i=1] X-Rspamd-Action: greylist X-Rspamd-Server: mail01.haj.ipfire.org Hi Michael Michael Tremer schreef op za 04-10-2025 om 11:52 [+0100]: > Hello Robin, >=20 > > On 2 Oct 2025, at 22:04, Robin Roevens > > wrote: > >=20 > > Hi Michael > >=20 > > I saw there is a suricata-reporter in the upcoming CU. And I was > > wondering if I could add an additional reporter into it for sending > > alerts straight to Zabbix, next to syslog and email. >=20 > I think that is a great idea! That makes at least 2 of us :-) >=20 > > I have already been experimenting with parsing fast.log using the > > zabbix_agentd, which seems to work quite well. But since there is > > now a > > reporter, it would be nice to have it support sending alerts to > > zabbix > > directly instead of zabbix separately monitoring the fast.log file. >=20 > Parsing log files like that is never fun. I also think that I > remember that suricata has deprecated support for writing fast.log, > but then it does not seem to be deprecated any more. Looking at the > alternative EVE format, fast.log is just there to have something that > is somewhat human-readable, but at the same time does not carry a lot > of information. I had considered EVE logs but it was not really clear to me how to distinguish normal events and actual alert events from than, and I saw that it is currently disabled, but that the reporter will soon enable it to connect to a socket. So I won't be able to listen in with the zabbix_agent.. I think..=C2=A0 Anyway, no longer relevant if it is build in into the reporter. =20 >=20 > > If that would be ok for you. There are 2 possible ways to do this: > > - using the zabbix_utils python library: > > https://blog.zabbix.com/python-zabbix-utils/27056/ > > - or using the zabbix_sender command utility that currently gets > > installed when installing zabbix_agentd >=20 > The IPS could generate a lot of alerts. Especially when there is > bursts in (malicious) traffic and I am already unhappy with calling > sendmail - but there I have no other choice. >=20 > The daemon initially had some worker daemons so that reading from > Suricata would be entirely decoupled from processing any messages. > However, Python=E2=80=99s multiprocessing capabilities are really bad and= so > now it is a threaded implementation. Python isn=E2=80=99t good at that > either, but when we are calling a subprocess the GIL is being > released and so we should not actually spend too much time in the > Python code itself. >=20 > This has to be considered when adding yet another output so that the > process does not get stalled when Zabbix does not respond, is > overwhelmed or anything else. We could simply combat that by adding a > couple more worker threads if necessary - but I am getting ahead of > myself. >=20 > So the Python module would be by far preferable. It seems to have a > nice API, but we will have to check how it works underneath to reach > the objectives outlined above. By default the zabbix sender functionality of the library has a timeout of 10s. When the Zabbix server is healthy, that timeout is never reached. But when unavailable or overloaded, that can happen. This timeout is configurable. It also has the possibility to bulk-send entries, which it will internally split into chunks of max 250 items which will then each be sent separately. Each chunk then has the 10s timout. Chunksize can be configured also. So to minimize zabbix communications, it could be a possibility to cache events within a timeframe (say 30s) and only once per 30s send all events in the queue. 30s is no longer 'real-time', so I would like to avoid that, but it may be a good idea to at least buffer within 1s to prevent possible hammering the server every millisecond in some extreme cases? There is also an async version of the sender functionality included in the library. I've not yet done any async python code, so I'm not sure I understand it correctly, but I assume it can make the actual sending a sort of background job, while the reporter can continue its other work? >=20 > > I assume, using the python library will probably be the most > > performant > > option; But then I should also create a zabbix_utils python library > > pak-file? >=20 > Hmm, this is where it is getting interesting. >=20 > The suricata-reporter stuff is supposed to be rather distro agnostic. > Besides Python itself, it only has a dependency to sqlite3 (which > normally comes with the Python standard library) and reportlab to > generate the PDFs. The goal is to run it on IPFire 2 for now, but it > would also be pretty much ready as is to run in IPFire 3 some day. Of > course it runs on any other distro - although I don=E2=80=99t know what it > would do on a non-Linux OS. >=20 > I would like to keep it lightweight and hopefully some more people > would adopt it. >=20 > I believe adding zabbix should be done as an optional dependency. We > could simply try to import the Zabbix modules. If they are there, the > process has support for Zabbix. If not, then not. That could simply > be handled at runtime. >=20 > That way, we could have zabbix as an add-on in IPFire instead of > adding it to the core distribution and if other people want to run > the whole thing, they can enjoy a minimum amount of dependencies and > don=E2=80=99t have to build/install any zabbix packages if they don=E2=80= =99t need > them. I agree that suricata-reporter in itself should be as lightweight and distro agnostic as possible. Adding zabbix as an optional dependency is an option. But I was wondering if it wouldn't be more elegant to have suricata-reporter support some sort of reporter modules or plugins, which could then contain the email, pdf and zabbix functionalities. That way, you could ship suricata-reporter without zabbix functionalities, and I could create a suricata-reporter-zabbix-plugin with the zabbix specific code in it that, when installed, is picked up by suricata-reporter. For IPFire I could then just include the reporter plugin in the zabbix_agentd pak on in a dependent separate pak which in turn depends on a new zabbix-utils pak containing the zabbix python library, and for other distro's, it can be downloaded from my git, and/or I add it to an ipfire git repo? >=20 > > Both the python module and the commandline cli have the possibility > > to > > get zabbix server connection info from the zabbix_agentd configfile > > so > > config of the reporter would be something like: > > [zabbix] > > enabled =3D true > > zabbix_agentd_config =3D /etc/zabbix_agentd/zabbix_agentd.conf > > alert_item_key =3D ipfire.suricata.event.get >=20 > I mean, this is all up to you. I would definitely do this bit: >=20 > [zabbix] > enabled =3D true >=20 > That way it is consistent with the remainder. Any other options, well > I don=E2=80=99t know what you would need :) I do enjoy a daemon that does= not > need a lot of configuration though and therefore can be run with only > a very small configuration as it relies on sensible defaults. I think what I gave as an example, could suffice. It may be usefull to support also explicit zabbix server settings instead of referencing the zabbix_agentd config. On IPFire, using the zabbix_agentd config is probably the best option to avoid duplicate configurations. On other distro's or use cases of the reporter, it may well be that it is used without a zabbix_agent installed, hence without a pre-existing config that can be used to get zabbix server details from. And in case of the plugin-architecture, I think easiest is to also support a loadable config-directory, so config files from other pak- files can be dropped in it? >=20 > > Then the reporter can format the incoming suricata alert/event as > > json > > and send it to the configured alert_item_key on the zabbix server > > as > > configured in the zabbix_agentd.conf >=20 > We do store the entire JSON object as it comes from Suricata in the > sqlite3 database. That way we have all the information that could > possible be available to us even though we don=E2=80=99t need everything > right now. The fast.log format definitely lacks a lot of important > information, but it is nicely readable. >=20 > Depending on what processing Zabbix can do later and what information > it permanently-ish stores, I would send the whole lot. The JSON > objects are small enough and over the wire they would compress > nicely, too. On the daemon side I wouldn=E2=80=99t implement this to be > configurable because that only makes everything more complicated. I will have to study what would exactly come through. But zabbix has a quite powerfull preprocessing engine which can format, validate, split, and can do almost anything with the incoming data. So that would probably indeed be the easiest method >=20 > > Is this something you are open to? Then I can try to create a patch > > for > > suricata-reporter. (where should I then submit it? Also on this > > list?) >=20 > Yes, very much so. >=20 > Patches can go on this list. >=20 > Let=E2=80=99s talk about any implementation details in advance. I feel a > little bit precious about this program because it is nice, shiny and > new and the code is very simple. I would like to keep it that way. And here I go suggesting wild extensible plugin frameworks :-) However, I don't think that should be very complicated. And it would create an ability to keep the reporter clean, and leave possible future extra plugins in the hands of others. =20 >=20 > > If not I will have to continue working on the fast.log parsing. >=20 > Don=E2=80=99t. That will only make you unhappy. Thanks.. I need some happiness once in a while :-) >=20 > > And while on the topic of monitoring suricata; I would like to get > > some > > extra stats from it, which, for as far as I currently know, can be > > retrieved using the suricata unix-socket that is currently disabled > > by > > default on ipfire. Many seem to use a 'suricatasc' tool to query > > suricata using that socket, but that tool is not available on > > ipfire. >=20 > What stats do you want/need? For starters I was thinking iface-list and iface-stat to get interface- specific stats about number of packets scanned, dropped, bypassed. But I will have to experiment a bit with it to see what stats can be valuable. >=20 > I am happy to turn on the socket and built the tool. Does it come > with the suricata package? I think it is a build option. I saw Adolf already proposed to enable it. >=20 > Please submit patches :) I think this could be done first before > diving into Zabbix because it should be a very simple change. >=20 > > Is it possible to have it on ipfire?, or should I start > > experimenting > > using socat? >=20 > No, if there are tools built already, use the code that is available. ok >=20 > > And if succesful, is it then allowed for a future zabbix_agentd > > addon > > pak to enable that socket in the suricata config? >=20 > Enable the socket by default. It won=E2=80=99t hurt anyone. ok.. most certainly it is better to have it enabled by default when the suricatasc tool is also available on IPFire. Giving users also the possibility to use it manually or to query suricatta using other tools than Zabbix. >=20 > > If you dislike the idea of enabling and querying the socket, > > another > > possibility is having suricata dump stats in a seperate stats.log > > which > > I should then be able to parse using Zabbix. >=20 > Hmm, but then we are back to parsing logs. That is never fun. agreed >=20 > > Before I start any implementations, what are your thoughts about > > all > > this ? >=20 > Very very happy with this here. >=20 > Let me know how I can help. >=20 > -Michael >=20 > > Regards > > Robin > >=20 > > --=20 > > Dit bericht is gescanned op virussen en andere gevaarlijke > > inhoud door MailScanner en lijkt schoon te zijn. > >=20 >=20 Regards Robin --=20 Dit bericht is gescanned op virussen en andere gevaarlijke inhoud door MailScanner en lijkt schoon te zijn.