From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: [PATCH] crontab: add periodic cleanup the collectd RRD (graphs) Date: Thu, 25 Aug 2022 16:44:23 +0100 Message-ID: In-Reply-To: <44F26D75-426C-45A8-A405-70A20BEBA807@gmail.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============5726143680339478406==" List-Id: --===============5726143680339478406== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello Jon, > On 25 Aug 2022, at 15:40, Jon Murphy wrote: >=20 > Hey Michael, >=20 > Over the years I=E2=80=99ve used four different IPFire boxes. And during a= "late-summer cleaning" I found lots of old RRD files. I ran this command:=20 > `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls` Was this just because you had a look around in the file system, or did you su= ffer from low disk space? > And among those were lots of old unused openvpn graphs. I think I had 8-10= old openvpn graphs that were more than 1 year old. >=20 > I had started to write a wiki page for users about finding and deleting old= `/var/log/rrd/collectd/localhost/openvpn-*` RRDs. And then the wiki grew to= ALL old graphs. =20 >=20 > Below is a snipped version of my original `find`. I think there were ~40 = total old obsolete RRD files. >=20 > ``` > [root(a)ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls > 1838 304 -rw-r--r-- 1 root root 307712 Nov 17 2019 /va= r/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd > 1794 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /va= r/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd > 1821 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /va= r/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd > . . . > 1831 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /va= r/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd > 1888 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /va= r/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd > 1785 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /va= r/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd > . . . > 1791 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /va= r/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd > 1880 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /va= r/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd > 1875 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /va= r/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd > 1874 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /va= r/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd > 1781 152 -rw-r--r-- 1 root root 154888 Jan 13 2020 /va= r/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd > 1937 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /va= r/log/rrd/iptraffic/192.168.60.105.rrd > 1938 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /va= r/log/rrd/iptraffic/192.168.60.147.rrd > 1935 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /va= r/log/rrd/iptraffic/192.168.60.194.rrd > [root(a)ipfire ~] #=20 > ``` >=20 > As you can see some of the files were left over from 2017 and they did not = get cleaned out during hardware changes. I obviously did not follow the migr= ations wiki! Well, I suppose if it doesn=E2=80=99t happen automatically, we cannot rely on= this :) I never follow this either. > The wiki page was to be instructions for users to `find` the files and then= to delete the files via the command line. While is was writing the wiki I r= emembered a Dev Mailing list message about sending users to the shell partial= ly defeats the purpose of IPFire (bad paraphrase!). So instead of writing a = wiki I wrote it a weekly fcronjob to the help everyone. > More than you ever wanted to know! Hope this helps! Yes it does, but in the end my concern remains: Is it not possible that an OpenVPN connection is only being used once a year = or something like that? Would we then not wipe any data? Wouldn=E2=80=99t tho= se databases come back again? I suppose a database that has not been touched in over a year would not be ca= rrying any information because that would have been rotated out of it (becaus= e that is how RRD databases work). So maybe that can go. But I would argue on= ly then. So we would have to check what the longest time is those databases r= etain any information. Personally, I would redesign this whole thing a little. I would make them sto= re a lot more data and I would even look for more metrics to collect. So, the= extra space for a RRD database wouldn=E2=80=99t really bother me, even if th= e whole directory is going to use up gigabytes. If that is a problem for othe= r users, I would like to know. Best, -Michael > Jon >=20 >=20 >> On Aug 25, 2022, at 5:48 AM, Michael Tremer = wrote: >>=20 >> Hello Jon, >>=20 >> Thanks for the patch. >>=20 >> Sorry if this sounds like a stupid question, but what is the rationale for= this? >>=20 >> -Michael >>=20 >>> On 25 Aug 2022, at 01:31, Jon Murphy wrote: >>>=20 >>> - Created (mostly) for old openvpn graphs >>> - RRD removed when no graph modification for +365 days >>> - chosen since graph max out is 365 days >>> - fcron job runs once per week >>> - chosen since this is just a cleanup and it doesnt need to run everyday >>> Note: logging can be added if needed. >>>=20 >>> Signed-off-by: Jon Murphy >>> --- >>> config/cron/crontab | 3 +++ >>> 1 file changed, 3 insertions(+) >>>=20 >>> diff --git a/config/cron/crontab b/config/cron/crontab >>> index b727531fc..7088e0b75 100644 >>> --- a/config/cron/crontab >>> +++ b/config/cron/crontab >>> @@ -80,5 +80,8 @@ HOME=3D/ >>> # Cleanup the mail spool directory >>> %weekly * * /usr/sbin/dma-cleanup-spool >>>=20 >>> +# Cleanup the collectd RRD (graphs) >>> +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -de= lete -o -type d -empty -delete >>> + >>> # Update DNS trust anchor >>> %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unb= ound/root.key -c /etc/unbound/icannbundle.pem >>> --=20 >>> 2.30.2 >>>=20 >>=20 >=20 --===============5726143680339478406==--