From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.haj.ipfire.org (localhost [IPv6:::1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4ctKpZ4kXVz331Q for ; Fri, 24 Oct 2025 11:06:30 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client CN "mail01.haj.ipfire.org", Issuer "R12" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4ctKpW2D83z2xQd for ; Fri, 24 Oct 2025 11:06:27 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4ctKp54kmFzlL; Fri, 24 Oct 2025 11:06:05 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1761303965; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pCUUFM0Wm3j//NgIwT1WdeFqS/qbv6fhkp6AXWoQmfU=; b=O80rKlpCs0i6r2wzT2+mEGek4tsNbZSTN1PDUkWgrP1Ocn+JieaWJpDIX07RckJR2XeGYg 0+l8K56YXtz6v1Aw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1761303965; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pCUUFM0Wm3j//NgIwT1WdeFqS/qbv6fhkp6AXWoQmfU=; b=FqQDJuDgtqII30FZXHiV5smgGPF9CdEAjhkyc9pcJufQuSIMPxFi0fv5w7k0QIuBDfolYW fYMBPuVFAAh7AdWbdknf5mgf/MnUXmFDMI5A9E2zLJ2xHtuSFNlI8NkEk7J16VMGg6T+AD JV2ly8OinjbREvVu5n3dBzgfhuUDvFe8KJKixxcEaDnWZt+SuB53EMWsEDSagU81if9LyR MKuFUTFZ3LvW6FKDfINf9NJI+D0FJj4ReDxCtDsXyMEwwOcIJ9rUD6wdi4dEtR6rDBJauw gSI3NU3yAOsIHtjVKij0cKHtReGH1EbT2kibbpY8f00RzdWPTZLDV9jHhy1FhA== Content-Type: text/plain; charset=utf-8 Precedence: list List-Id: List-Subscribe: , List-Unsubscribe: , List-Post: List-Help: Sender: Mail-Followup-To: Mime-Version: 1.0 Subject: Re: [SPAM Warning!]Re: Zabbix support for suricata-reporter From: Michael Tremer In-Reply-To: Date: Fri, 24 Oct 2025 12:06:05 +0100 Cc: "IPFire: Development-List" Content-Transfer-Encoding: quoted-printable Message-Id: <17F5618D-343E-4A17-AA60-1BA5B9BA6806@ipfire.org> References: <56ad0ff847bb0b5b5808084321ac52efd63cd16f.camel@disroot.org> <708978AA-034A-4533-BF40-257A843AFF8E@ipfire.org> To: Robin Roevens Hello Robin, > On 16 Oct 2025, at 22:59, Robin Roevens = wrote: >=20 > Hi Michael >=20 > Michael Tremer schreef op za 04-10-2025 om 11:52 [+0100]: >> Hello Robin, >>=20 >>> On 2 Oct 2025, at 22:04, Robin Roevens >>> wrote: >>>=20 >>> Hi Michael >>>=20 >>> I saw there is a suricata-reporter in the upcoming CU. And I was >>> wondering if I could add an additional reporter into it for sending >>> alerts straight to Zabbix, next to syslog and email. >>=20 >> I think that is a great idea! > That makes at least 2 of us :-) I am sure there are more. On Reddit, your Zabbix add-on seems to be very = popular, too: = https://www.reddit.com/r/ipfire/comments/1oapu1t/ipfire_and_zabbix_integra= tion/ >>=20 >>> I have already been experimenting with parsing fast.log using the >>> zabbix_agentd, which seems to work quite well. But since there is >>> now a >>> reporter, it would be nice to have it support sending alerts to >>> zabbix >>> directly instead of zabbix separately monitoring the fast.log file. >>=20 >> Parsing log files like that is never fun. I also think that I >> remember that suricata has deprecated support for writing fast.log, >> but then it does not seem to be deprecated any more. Looking at the >> alternative EVE format, fast.log is just there to have something that >> is somewhat human-readable, but at the same time does not carry a lot >> of information. > I had considered EVE logs but it was not really clear to me how to > distinguish normal events and actual alert events from than, and I saw > that it is currently disabled, but that the reporter will soon enable > it to connect to a socket. So I won't be able to listen in with the > zabbix_agent.. I think..=20 > Anyway, no longer relevant if it is build in into the reporter. Yes, this is kind of abstracted away for you. The problem with parsing a = log file is also where to resume after a restart of the service? What = happens when the log file is being rotated? Therefore I have chosen to write daemon that suricata is directly = writing to. This comes with some other challenges, but I thought that it = would be much easier to solve these instead. We are receiving the full EVE log and we can process any kind of events = that we are interested in. Currently, this is only the alerts. >>=20 >>> If that would be ok for you. There are 2 possible ways to do this: >>> - using the zabbix_utils python library: >>> https://blog.zabbix.com/python-zabbix-utils/27056/ >>> - or using the zabbix_sender command utility that currently gets >>> installed when installing zabbix_agentd >>=20 >> The IPS could generate a lot of alerts. Especially when there is >> bursts in (malicious) traffic and I am already unhappy with calling >> sendmail - but there I have no other choice. >>=20 >> The daemon initially had some worker daemons so that reading from >> Suricata would be entirely decoupled from processing any messages. >> However, Python=E2=80=99s multiprocessing capabilities are really bad = and so >> now it is a threaded implementation. Python isn=E2=80=99t good at = that >> either, but when we are calling a subprocess the GIL is being >> released and so we should not actually spend too much time in the >> Python code itself. >>=20 >> This has to be considered when adding yet another output so that the >> process does not get stalled when Zabbix does not respond, is >> overwhelmed or anything else. We could simply combat that by adding a >> couple more worker threads if necessary - but I am getting ahead of >> myself. >>=20 >> So the Python module would be by far preferable. It seems to have a >> nice API, but we will have to check how it works underneath to reach >> the objectives outlined above. > By default the zabbix sender functionality of the library has a = timeout > of 10s. When the Zabbix server is healthy, that timeout is never > reached. But when unavailable or overloaded, that can happen. This > timeout is configurable. We would have to factor this in though. It must be possible for Zabbix = to be slow because it either is busy, or simply not easily reachable = over the network. A classic case could be where the firewall is under = some (D)DoS attack and we still don=E2=80=99t want to simply drop any = kind of alert because frankly, alerting the admin is exactly what this = service is for. > It also has the possibility to bulk-send entries, which it will > internally split into chunks of max 250 items which will then each be > sent separately. Each chunk then has the 10s timout. Chunksize can be > configured also. Sending entries in bulk would be great to reduce overhead. I don=E2=80=99t= know about Zabbix in particular, but often client libraries like this = use Python requests which will open up a new TCP/TLS connection per = request. That has a huge amount of overhead when you have hundreds of = alerts. If we would create some queue where everything that is interesting is = being called and try to once a second flush everything that is in the = queue to Zabbix, we should be able to batch a lot of alerts and (unless = we exceed the limit of 250 alerts per second) send them over one single = TCP connection per second. That would be very efficient even with large = numbers of alerts. Since this software is currently very young and since I have not seen it = being deployed in really large environments, I don=E2=80=99t quite know = what the target should be. I would consider it a possibility of some = people having a sustained rate of thousands of alerts per second. = Hopefully nobody will be trying to send this via email and more, but the = SQLite database can easily ingest this and therefore we can report on it = well. > So to minimize zabbix communications, it could be a possibility to > cache events within a timeframe (say 30s) and only once per 30s send > all events in the queue. 30s is no longer 'real-time', so I would like > to avoid that, but it may be a good idea to at least buffer within 1s > to prevent possible hammering the server every millisecond in some > extreme cases? Yes, absolutely. And although I don=E2=80=99t like threads in Python, I suppose this = should be implemented in a separate thread. If we build this well, when = the Zabbix client is waiting, it won=E2=80=99t block the GIL at all. > There is also an async version of the sender functionality included in > the library. I've not yet done any async python code, so I'm not sure = I > understand it correctly, but I assume it can make the actual sending a > sort of background job, while the reporter can continue its other = work? That would be an option, too. I like to write asynchronous Python, but = it makes things easily a lot more complex. Therefore, the reporter has = an async main thread and forks some worker threads which are implemented = synchronously. We could easily change the latter to be fully async. It = wouldn=E2=80=99t hurt me. I just didn=E2=80=99t have the need to do it. This would however allow us to dispatch multiple sets of alerts at the = same time and we wouldn=E2=80=99t have to wait for one request to be = completed. This could then have the potential to send a lot of data = really fast to the Zabbix server and I am not sure how that would cope. = Do we have some idea how long sending a single alert would take, a = hundred alerts? >>> I assume, using the python library will probably be the most >>> performant >>> option; But then I should also create a zabbix_utils python library >>> pak-file? >>=20 >> Hmm, this is where it is getting interesting. >>=20 >> The suricata-reporter stuff is supposed to be rather distro agnostic. >> Besides Python itself, it only has a dependency to sqlite3 (which >> normally comes with the Python standard library) and reportlab to >> generate the PDFs. The goal is to run it on IPFire 2 for now, but it >> would also be pretty much ready as is to run in IPFire 3 some day. Of >> course it runs on any other distro - although I don=E2=80=99t know = what it >> would do on a non-Linux OS. >>=20 >> I would like to keep it lightweight and hopefully some more people >> would adopt it. >>=20 >> I believe adding zabbix should be done as an optional dependency. We >> could simply try to import the Zabbix modules. If they are there, the >> process has support for Zabbix. If not, then not. That could simply >> be handled at runtime. >>=20 >> That way, we could have zabbix as an add-on in IPFire instead of >> adding it to the core distribution and if other people want to run >> the whole thing, they can enjoy a minimum amount of dependencies and >> don=E2=80=99t have to build/install any zabbix packages if they = don=E2=80=99t need >> them. >=20 > I agree that suricata-reporter in itself should be as lightweight and > distro agnostic as possible. Adding zabbix as an optional dependency = is > an option. But I was wondering if it wouldn't be more elegant to have > suricata-reporter support some sort of reporter modules or plugins, > which could then contain the email, pdf and zabbix functionalities. Hmm, it kind of is modular. There is a function that is being called = whenever an alert has been received and depending on configuration, this = function calls the functions to send emails, syslog, etc. = https://git.ipfire.org/?p=3Dsuricata-reporter.git;a=3Dblob;f=3Dsrc/suricat= a-reporter.in;h=3De1c25b48b49f7c6818aadfc8ad80439f5c35230b;hb=3DHEAD#l427 We could in theory create something here where modules register a = function which will be called. Not a problem. For simplicity, and to have some quick result in the first place, I have = chosen this approach. See where the challenges are and then make the = code more expandable. > That way, you could ship suricata-reporter without zabbix > functionalities, and I could create a suricata-reporter-zabbix-plugin > with the zabbix specific code in it that, when installed, is picked up > by suricata-reporter. For IPFire I could then just include the = reporter > plugin in the zabbix_agentd pak on in a dependent separate pak which = in > turn depends on a new zabbix-utils pak containing the zabbix python > library, and for other distro's, it can be downloaded from my git, > and/or I add it to an ipfire git repo? This is where it will probably get messy. As mentioned before, the = performance of this tool lies in not blocking the process. If we would = have externally maintained modules, we would not have control over this = and people might add some stuff that harms the performance of the = software. Arguably that won=E2=80=99t be our problem then, but why make = people=E2=80=99s lives hard? I therefore would prefer to keep all the code in one repository so that = we have control over it. It is all still very young and we want to have = the option to make changes (even big ones) if we need to. For example = make the entire software asynchronous instead of having synchronous = worker threads. External modules would have to keep up with those = changes, and since open source development has very sadly for me turned = into building my own things on my own, and other people doing the same = with usually none (and if they really have to minimal) conversation, we = would probably just keep breaking things or feeling forced to keep = everything as is. I would say that we should integrate Zabbix into the main project and if = there are is really a large number of other people coming to add more = plugins, we can still add external modules. That way, we would at least = have matured the software a little bit and be able to offer a more = stable API. Right now, nobody but us is using this software and so I don=E2=80=99t = want to accommodate needs of people I don=E2=80=99t even know yet. What = could they possibly need? I don=E2=80=99t know. Building out every = possible option under the sun for zero users is not a good investment of = anyone=E2=80=99s time. >>=20 >>> Both the python module and the commandline cli have the possibility >>> to >>> get zabbix server connection info from the zabbix_agentd configfile >>> so >>> config of the reporter would be something like: >>> [zabbix] >>> enabled =3D true >>> zabbix_agentd_config =3D /etc/zabbix_agentd/zabbix_agentd.conf >>> alert_item_key =3D ipfire.suricata.event.get >>=20 >> I mean, this is all up to you. I would definitely do this bit: >>=20 >> [zabbix] >> enabled =3D true >>=20 >> That way it is consistent with the remainder. Any other options, well >> I don=E2=80=99t know what you would need :) I do enjoy a daemon that = does not >> need a lot of configuration though and therefore can be run with only >> a very small configuration as it relies on sensible defaults. > I think what I gave as an example, could suffice. It may be usefull to > support also explicit zabbix server settings instead of referencing = the > zabbix_agentd config. On IPFire, using the zabbix_agentd config is > probably the best option to avoid duplicate configurations. On other > distro's or use cases of the reporter, it may well be that it is used > without a zabbix_agent installed, hence without a pre-existing config > that can be used to get zabbix server details from. I am flexible here. We are using Python=E2=80=99s configparser and that = could either take a filename (or have a default), or simple take server, = username, password or any other kind of credential to contact the = service. > And in case of the plugin-architecture, I think easiest is to also > support a loadable config-directory, so config files from other pak- > files can be dropped in it? We can do this anyways. Could you please create a bug report for this so = that we track this? It is not too urgent to implement, but whenever I = have time, I am happy to do it. https://bugzilla.ipfire.org/enter_bug.cgi?product=3DSuricata%20Reporter >>=20 >>> Then the reporter can format the incoming suricata alert/event as >>> json >>> and send it to the configured alert_item_key on the zabbix server >>> as >>> configured in the zabbix_agentd.conf >>=20 >> We do store the entire JSON object as it comes from Suricata in the >> sqlite3 database. That way we have all the information that could >> possible be available to us even though we don=E2=80=99t need = everything >> right now. The fast.log format definitely lacks a lot of important >> information, but it is nicely readable. >>=20 >> Depending on what processing Zabbix can do later and what information >> it permanently-ish stores, I would send the whole lot. The JSON >> objects are small enough and over the wire they would compress >> nicely, too. On the daemon side I wouldn=E2=80=99t implement this to = be >> configurable because that only makes everything more complicated. >=20 > I will have to study what would exactly come through. But zabbix has a > quite powerfull preprocessing engine which can format, validate, = split, > and can do almost anything with the incoming data. So that would > probably indeed be the easiest method We could of course do some processing in the reporter, or send the raw = stuff. Up to you. It could even be configurable. But before we figure that out, would you like to package the Zabbix = Python modules that we need for IPFire, or are they already part of the = zabbix agent package? >>=20 >>> Is this something you are open to? Then I can try to create a patch >>> for >>> suricata-reporter. (where should I then submit it? Also on this >>> list?) >>=20 >> Yes, very much so. >>=20 >> Patches can go on this list. >>=20 >> Let=E2=80=99s talk about any implementation details in advance. I = feel a >> little bit precious about this program because it is nice, shiny and >> new and the code is very simple. I would like to keep it that way. > And here I go suggesting wild extensible plugin frameworks :-) > However, I don't think that should be very complicated. And it would > create an ability to keep the reporter clean, and leave possible = future > extra plugins in the hands of others. It could. If those =E2=80=9Cothers=E2=80=9D turn up, let=E2=80=99s build = it. If not, we won=E2=80=99t need it. >>=20 >>> If not I will have to continue working on the fast.log parsing. >>=20 >> Don=E2=80=99t. That will only make you unhappy. > Thanks.. I need some happiness once in a while :-) >>=20 >>> And while on the topic of monitoring suricata; I would like to get >>> some >>> extra stats from it, which, for as far as I currently know, can be >>> retrieved using the suricata unix-socket that is currently disabled >>> by >>> default on ipfire. Many seem to use a 'suricatasc' tool to query >>> suricata using that socket, but that tool is not available on >>> ipfire. >>=20 >> What stats do you want/need? > For starters I was thinking iface-list and iface-stat to get = interface- > specific stats about number of packets scanned, dropped, bypassed. But > I will have to experiment a bit with it to see what stats can be > valuable. This is a bit of a side note, but it has tickled my curiosity as I am = currently working on replacing collectd. I want something that is more = flexible, more custom to us and allows us to collect a lot more = =E2=80=9Cinteresting=E2=80=9D data. Whatever that would be. Out of =E2=80=9Cfun=E2=80=9D I wrote a little suricata plugin (yes, it = has a little plugin thing): = https://git.ipfire.org/?p=3Dcollecty.git;a=3Dblob;f=3Dsrc/daemon/sources/s= uricata.c;h=3De317566a5d0ad0fa36ef97542e145ee68314176b;hb=3DHEAD It calls suricatasc --command=3Ddump-counters (asynchronously in C) and = parses the returned JSON. But then I didn=E2=80=99t quite know what I = should actually store?! I would be happy to hear your ideas. But this is probably another = conversation. >>=20 >> I am happy to turn on the socket and built the tool. Does it come >> with the suricata package? > I think it is a build option. I saw Adolf already proposed to enable > it. Yes, it is enabled and I have tested it. It returns a whole tapestry = with metrics. >=20 >>=20 >> Please submit patches :) I think this could be done first before >> diving into Zabbix because it should be a very simple change. >>=20 >>> Is it possible to have it on ipfire?, or should I start >>> experimenting >>> using socat? >>=20 >> No, if there are tools built already, use the code that is available. > ok >=20 >>=20 >>> And if succesful, is it then allowed for a future zabbix_agentd >>> addon >>> pak to enable that socket in the suricata config? >>=20 >> Enable the socket by default. It won=E2=80=99t hurt anyone. > ok.. most certainly it is better to have it enabled by default when = the > suricatasc tool is also available on IPFire. Giving users also the > possibility to use it manually or to query suricatta using other tools > than Zabbix. Yes, this does not cost anything extra. It is done. >>=20 >>> If you dislike the idea of enabling and querying the socket, >>> another >>> possibility is having suricata dump stats in a seperate stats.log >>> which >>> I should then be able to parse using Zabbix. >>=20 >> Hmm, but then we are back to parsing logs. That is never fun. > agreed >>=20 >>> Before I start any implementations, what are your thoughts about >>> all >>> this ? >>=20 >> Very very happy with this here. >>=20 >> Let me know how I can help. >>=20 >> -Michael >>=20 >>> Regards >>> Robin >>>=20 >>> --=20 >>> Dit bericht is gescanned op virussen en andere gevaarlijke >>> inhoud door MailScanner en lijkt schoon te zijn. >>>=20 >>=20 > Regards > Robin >=20 > --=20 > Dit bericht is gescanned op virussen en andere gevaarlijke > inhoud door MailScanner en lijkt schoon te zijn. >=20