Re: [PATCH] crontab: add periodic cleanup the collectd RRD (graphs)

[Michael Tremer <michael.tremer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 6091 bytes --]

Hello Jon,

> On 25 Aug 2022, at 15:40, Jon Murphy <jcmurphy26(a)gmail.com> wrote:
> 
> Hey Michael,
> 
> Over the years I’ve used four different IPFire boxes.  And during a "late-summer cleaning" I found lots of old RRD files.  I ran this command: 
> `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls`

Was this just because you had a look around in the file system, or did you suffer from low disk space?

> And among those were lots of old unused openvpn graphs.  I think I had 8-10 old openvpn graphs that were more than 1 year old.
> 
> I had started to write a wiki page for users about finding and deleting old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs.  And then the wiki grew to ALL old graphs.  
> 
> Below is a snipped version of my original `find`.   I think there were ~40 total old obsolete RRD files.
> 
> ```
> [root(a)ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls
>      1838    304 -rw-r--r--   1 root     root       307712 Nov 17  2019 /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd
>      1794    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd
>      1821    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd
> . . .
>      1831    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd
>      1888    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd
>      1785    152 -rw-r--r--   1 root     root       154888 Aug 12  2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd
> . . .
>      1791    152 -rw-r--r--   1 root     root       154888 Aug 12  2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd
>      1880    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd
>      1875    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd
>      1874    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd
>      1781    152 -rw-r--r--   1 root     root       154888 Jan 13  2020 /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd
>      1937     56 -rw-r--r--   1 nobody   nobody      56800 Jul 18  2021 /var/log/rrd/iptraffic/192.168.60.105.rrd
>      1938     56 -rw-r--r--   1 nobody   nobody      56800 Jul 18  2021 /var/log/rrd/iptraffic/192.168.60.147.rrd
>      1935     56 -rw-r--r--   1 nobody   nobody      56800 Jul 18  2021 /var/log/rrd/iptraffic/192.168.60.194.rrd
> [root(a)ipfire ~] # 
> ```
> 
> As you can see some of the files were left over from 2017 and they did not get cleaned out during hardware changes.  I obviously did not follow the migrations wiki!

Well, I suppose if it doesn’t happen automatically, we cannot rely on this :) I never follow this either.

> The wiki page was to be instructions for users to `find` the files and then to delete the files via the command line.  While is was writing the wiki I remembered a Dev Mailing list message about sending users to the shell partially defeats the purpose of IPFire (bad paraphrase!).  So instead of writing a wiki I wrote it a weekly fcronjob to the help everyone.

> More than you ever wanted to know!  Hope this helps!

Yes it does, but in the end my concern remains:

Is it not possible that an OpenVPN connection is only being used once a year or something like that? Would we then not wipe any data? Wouldn’t those databases come back again?

I suppose a database that has not been touched in over a year would not be carrying any information because that would have been rotated out of it (because that is how RRD databases work). So maybe that can go. But I would argue only then. So we would have to check what the longest time is those databases retain any information.

Personally, I would redesign this whole thing a little. I would make them store a lot more data and I would even look for more metrics to collect. So, the extra space for a RRD database wouldn’t really bother me, even if the whole directory is going to use up gigabytes. If that is a problem for other users, I would like to know.

Best,
-Michael

> Jon
> 
> 
>> On Aug 25, 2022, at 5:48 AM, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>> 
>> Hello Jon,
>> 
>> Thanks for the patch.
>> 
>> Sorry if this sounds like a stupid question, but what is the rationale for this?
>> 
>> -Michael
>> 
>>> On 25 Aug 2022, at 01:31, Jon Murphy <jon.murphy(a)ipfire.org> wrote:
>>> 
>>> - Created (mostly) for old openvpn graphs
>>> - RRD removed when no graph modification for +365 days
>>> - chosen since graph max out is 365 days
>>> - fcron job runs once per week
>>> - chosen since this is just a cleanup and it doesnt need to run everyday
>>> Note: logging can be added if needed.
>>> 
>>> Signed-off-by: Jon Murphy <jon.murphy(a)ipfire.org>
>>> ---
>>> config/cron/crontab | 3 +++
>>> 1 file changed, 3 insertions(+)
>>> 
>>> diff --git a/config/cron/crontab b/config/cron/crontab
>>> index b727531fc..7088e0b75 100644
>>> --- a/config/cron/crontab
>>> +++ b/config/cron/crontab
>>> @@ -80,5 +80,8 @@ HOME=/
>>> # Cleanup the mail spool directory
>>> %weekly * * /usr/sbin/dma-cleanup-spool
>>> 
>>> +# Cleanup the collectd RRD (graphs)
>>> +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete
>>> +
>>> # Update DNS trust anchor
>>> %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem
>>> -- 
>>> 2.30.2
>>> 
>> 
>