Re: [PATCH] crontab: add periodic cleanup the collectd RRD (graphs)

[Michael Tremer <michael.tremer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 7373 bytes --]

Hello,

> On 25 Aug 2022, at 17:17, Jon Murphy <jcmurphy26(a)gmail.com> wrote:
> 
> Michael,
> 
> 
>> On Aug 25, 2022, at 10:44 AM, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>> 
>> Hello Jon,
>> 
>>> On 25 Aug 2022, at 15:40, Jon Murphy <jcmurphy26(a)gmail.com> wrote:
>>> 
>>> Hey Michael,
>>> 
>>> Over the years I’ve used four different IPFire boxes.  And during a "late-summer cleaning" I found lots of old RRD files.  I ran this command: 
>>> `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls`
>> 
>> Was this just because you had a look around in the file system, or did you suffer from low disk space?
> 
> Just looking…

:)

> 
>> 
>>> And among those were lots of old unused openvpn graphs.  I think I had 8-10 old openvpn graphs that were more than 1 year old.
>>> 
>>> I had started to write a wiki page for users about finding and deleting old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs.  And then the wiki grew to ALL old graphs.  
>>> 
>>> Below is a snipped version of my original `find`.   I think there were ~40 total old obsolete RRD files.
>>> 
>>> ```
>>> [root(a)ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls
>>>     1838    304 -rw-r--r--   1 root     root       307712 Nov 17  2019 /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd
>>>     1794    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd
>>>     1821    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd
>>> . . .
>>>     1831    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd
>>>     1888    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd
>>>     1785    152 -rw-r--r--   1 root     root       154888 Aug 12  2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd
>>> . . .
>>>     1791    152 -rw-r--r--   1 root     root       154888 Aug 12  2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd
>>>     1880    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd
>>>     1875    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd
>>>     1874    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd
>>>     1781    152 -rw-r--r--   1 root     root       154888 Jan 13  2020 /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd
>>>     1937     56 -rw-r--r--   1 nobody   nobody      56800 Jul 18  2021 /var/log/rrd/iptraffic/192.168.60.105.rrd
>>>     1938     56 -rw-r--r--   1 nobody   nobody      56800 Jul 18  2021 /var/log/rrd/iptraffic/192.168.60.147.rrd
>>>     1935     56 -rw-r--r--   1 nobody   nobody      56800 Jul 18  2021 /var/log/rrd/iptraffic/192.168.60.194.rrd
>>> [root(a)ipfire ~] # 
>>> ```
>>> 
>>> As you can see some of the files were left over from 2017 and they did not get cleaned out during hardware changes.  I obviously did not follow the migrations wiki!
>> 
>> Well, I suppose if it doesn’t happen automatically, we cannot rely on this :) I never follow this either.
>> 
>>> The wiki page was to be instructions for users to `find` the files and then to delete the files via the command line.  While is was writing the wiki I remembered a Dev Mailing list message about sending users to the shell partially defeats the purpose of IPFire (bad paraphrase!).  So instead of writing a wiki I wrote it a weekly fcronjob to the help everyone.
>> 
>>> More than you ever wanted to know!  Hope this helps!
>> 
>> Yes it does, but in the end my concern remains:
>> 
>> Is it not possible that an OpenVPN connection is only being used once a year or something like that? Would we then not wipe any data? Wouldn’t those databases come back again?
> 
> I will test by wiping the current data and then connecting via OpenVPN to see if the databases and graphs recover.

I am quite confident that should work just fine.

>> 
>> I suppose a database that has not been touched in over a year would not be carrying any information because that would have been rotated out of it (because that is how RRD databases work). So maybe that can go. But I would argue only then. So we would have to check what the longest time is those databases retain any information.
>> 
> 
> Who is the RRD guru for IPFire?  
> 
> I think there is an RRD dump command.  I will research.

  rrdtool dump /var/log/rrd/collectd/localhost/entropy/entropy.rrd

That will show you a looooong list with all the data points.

>> Personally, I would redesign this whole thing a little. I would make them store a lot more data and I would even look for more metrics to collect. So, the extra space for a RRD database wouldn’t really bother me, even if the whole directory is going to use up gigabytes. If that is a problem for other users, I would like to know.
> 
> OK on my side!  I am at 48% (Use%) for a 16 GB drive.

Yes, 16GB is perfectly fine.

> I’d like to see a periodic bandwidth test (speedtest).  Once in awhile I drop to near 0 Mbps bandwidth.  Then it is time to restart the cable gateway and maybe the local network.

Yeah, that would be really nice to have :) I am just not sure how much traffic all IPFire devices like that would create.

Best,
-Michael

> 
> 
> Jon
> 
>> 
>> Best,
>> -Michael
>> 
>>> Jon
>>> 
>>> 
>>>> On Aug 25, 2022, at 5:48 AM, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>> 
>>>> Hello Jon,
>>>> 
>>>> Thanks for the patch.
>>>> 
>>>> Sorry if this sounds like a stupid question, but what is the rationale for this?
>>>> 
>>>> -Michael
>>>> 
>>>>> On 25 Aug 2022, at 01:31, Jon Murphy <jon.murphy(a)ipfire.org> wrote:
>>>>> 
>>>>> - Created (mostly) for old openvpn graphs
>>>>> - RRD removed when no graph modification for +365 days
>>>>> - chosen since graph max out is 365 days
>>>>> - fcron job runs once per week
>>>>> - chosen since this is just a cleanup and it doesnt need to run everyday
>>>>> Note: logging can be added if needed.
>>>>> 
>>>>> Signed-off-by: Jon Murphy <jon.murphy(a)ipfire.org>
>>>>> ---
>>>>> config/cron/crontab | 3 +++
>>>>> 1 file changed, 3 insertions(+)
>>>>> 
>>>>> diff --git a/config/cron/crontab b/config/cron/crontab
>>>>> index b727531fc..7088e0b75 100644
>>>>> --- a/config/cron/crontab
>>>>> +++ b/config/cron/crontab
>>>>> @@ -80,5 +80,8 @@ HOME=/
>>>>> # Cleanup the mail spool directory
>>>>> %weekly * * /usr/sbin/dma-cleanup-spool
>>>>> 
>>>>> +# Cleanup the collectd RRD (graphs)
>>>>> +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete
>>>>> +
>>>>> # Update DNS trust anchor
>>>>> %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem
>>>>> -- 
>>>>> 2.30.2