From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.haj.ipfire.org (localhost [IPv6:::1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4dVtVy1fm6z2ywD for ; Tue, 16 Dec 2025 10:30:50 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [IPv6:2001:678:b28::25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client CN "mail01.haj.ipfire.org", Issuer "R12" (verified OK)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4dVtVh1pbvz2xMw for ; Tue, 16 Dec 2025 10:30:36 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail01.ipfire.org (Postfix) with ESMTPSA id 4dVtVZ2TfRzBY; Tue, 16 Dec 2025 10:30:30 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1765881030; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=e6F6sX9zIREnpzeY3dGbCN9KpRoP09GqGuWYf7Y/NxQ=; b=M3JAAy8uO7uS/djMXC4Yrot4nQ0XcRRyMFgO5EGYLhXVw6quVzMwJCxaDSglEls6XMLDi5 88TNxYQHkp3qAdCA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1765881030; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=e6F6sX9zIREnpzeY3dGbCN9KpRoP09GqGuWYf7Y/NxQ=; b=ToLkESEia+ndWiZSVmzkk1BQnucy/bJN6h85RTy3X9oi3ZfyGaL6/86FsrEJgJ2+g+sx7I pQZyk8kz9N5cNJxHPY9Jpa8q9L+HvXoeD121nOU9iTwf17h84kUHnFt2xfQ/OHkxoBuz/t IDTLfQrXD4ZqRS/zkrwy9KiLS8p7gw9csn818YUhz2/Ciz2sCVidF117SQ8ten1SJ/2WqP sG6JS4IuSnx7tiOCYS1v5bVMgdrdGS4aEPeSEgvOqM61pAtjCrSY6Yc3JCcF147ARjv9v1 BO9Dhw5YVzc/mZp6+SNYLDZDXuFobcx7e+o7ip7YpBH0v2Fe1fN44H4md9hCJA== Content-Type: text/plain; charset=utf-8 Precedence: list List-Id: List-Subscribe: , List-Unsubscribe: , List-Post: List-Help: Sender: Mail-Followup-To: Mime-Version: 1.0 Subject: Re: Large Suricata cache directory. From: Michael Tremer In-Reply-To: Date: Tue, 16 Dec 2025 10:30:29 +0000 Cc: Development Content-Transfer-Encoding: quoted-printable Message-Id: References: <8ac70e7aa3d72cf2de5cda09a52f1cf6@ipfire.org> <8BFE2D8F-E49A-4823-8795-0A6B54D6A1B5@ipfire.org> To: Adam Gibbons Hello Adam, > On 15 Dec 2025, at 19:29, Adam Gibbons = wrote: >=20 > Hi Michael, Adolf, >=20 > Yes, Adolf has patched the second issue. I=E2=80=99ve tested this and = backups dropped from 826 MB to around 10 MB. Thank you, Adolf, for that. This sounds like a very reasonable size for a backup file. > If the upstream Suricata fix lands soon, we may not need to do = anything further. Regarding the proposed `find` command, my only concern = is partial cache removal. When I removed the entire cache, Suricata = regenerated it cleanly on startup. I=E2=80=99m not sure how it behaves = when only some of the cache is missing, as I=E2=80=99ve only tested = removing all of it (`rm -rf /var/cache/suricata/sgh/*`). Perhaps it = would be cleaner and potentially safer to just purge the cache entirely? I suppose that Suricata will just re-generate anything that is missing = from the cache. It would just take a couple of seconds at startup. You can simply test this be removing half the files and restart = Suricata. If it fails to come up, I would consider this a bug that we = should be reporting upstream. However, that would surprise me if it was = implemented as such. -Michael > Thanks, > Adam >=20 >=20 > On 15 December 2025 16:54:51 GMT, Michael Tremer = wrote: > Hello Adam, >=20 > Thank you for raising this here. >=20 > We seem to have two different issues as far as I can see: >=20 > 1) The directory just keeps growing >=20 > 2) It is being backed up and completely blows up the size of the = backup >=20 > No. 2 has been fixed by Adolf. It is however quite interesting that we = already have something in /var/cache in the backup. The intention was to = have a valid set of rules available as soon as a backup is being = restored, but I think there is very little value in this. The rules are = probably long expires and will be re-downloaded again. >=20 > We also have a large number of other (also large in disk size) lists = around that we are not backing up, so I would propose that we remove = /var/cache/suricata from the backup entirely. Until we have made a = decision on this, I have merged Adolf=E2=80=99s patch. >=20 > Regarding No. 1, this is indeed a problem that Suricata does not clean = this up itself. We could add a simple command that looks a bit like = this: >=20 > find /var/cache/suricata/ -type f -atime +7 -delete >=20 > This would delete all of the cached files that have not been accessed = in the last seven days. >=20 > On my system, the entire directory is 1.4 GiB in size and the command = would remove 500 MiB. >=20 > Happy to read your thoughts. >=20 > -Michael >=20 > On 12 Dec 2025, at 16:49, Adam Gibbons = wrote: >=20 > Hi all, >=20 > As discussed on the forum > https://community.ipfire.org/t/re-large-backupfile/15346 > it appears that Suricata=E2=80=99s new cache optimisation feature is = creating a large number of files under > `/var/cache/suricata/sgh/`, which in some cases causes backup files to = grow to 800+ MB. >=20 > @Adolf has confirmed that this directory probably should not be = included in backups, as it is automatically regenerated, and I believe = he mentioned he is working on a patch to exclude it from the backup. >=20 > However, in the meantime, this directory continues to grow over time. = The upstream Suricata patches to automatically clean or maintain the = cache have not yet been merged, although they may be soon: >=20 > https://github.com/OISF/suricata/pull/13850 > https://github.com/OISF/suricata/pull/14400 >=20 > To me this represents a disk-space exhaustion risk on systems with = limited storage. Perhaps we should consider disabling Suricata=E2=80=99s = new cache optimisation feature until automatic cache cleanup/maintenance = is available upstream and included. >=20 > Thanks, > Adam >=20 >=20 >=20