Re: [PATCH] crontab: add periodic cleanup the collectd RRD (graphs)

[Michael Tremer <michael.tremer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 10517 bytes --]

I think this is fine for me:

Reviewed-by: Michael Tremer <michael.tremer(a)ipfire.org>

Thanks for the nudge.

-Michael

> On 20 Sep 2022, at 03:28, Jon Murphy <jcmurphy26(a)gmail.com> wrote:
> 
> Bump / nudge...    Is this OK?
> 
> 
> Jon
> 
> 
>> On Aug 25, 2022, at 4:34 PM, Jon Murphy <jcmurphy26(a)gmail.com> wrote:
>> 
>> Michael,
>> 
>> Last item below…
>> 
>> If OK please add your "Acked-by:" or "Reviewed-by:".
>> 
>> Jon
>> 
>>> On Aug 25, 2022, at 11:47 AM, Jon Murphy <jcmurphy26(a)gmail.com> wrote:
>>> 
>>> Michael,
>>> 
>>>> On Aug 25, 2022, at 11:32 AM, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>> 
>>>> Hello,
>>>> 
>>>>> On 25 Aug 2022, at 17:17, Jon Murphy <jcmurphy26(a)gmail.com> wrote:
>>>>> 
>>>>> Michael,
>>>>> 
>>>>> 
>>>>>> On Aug 25, 2022, at 10:44 AM, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>>> 
>>>>>> Hello Jon,
>>>>>> 
>>>>>>> On 25 Aug 2022, at 15:40, Jon Murphy <jcmurphy26(a)gmail.com> wrote:
>>>>>>> 
>>>>>>> Hey Michael,
>>>>>>> 
>>>>>>> Over the years I’ve used four different IPFire boxes.  And during a "late-summer cleaning" I found lots of old RRD files.  I ran this command: 
>>>>>>> `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls`
>>>>>> 
>>>>>> Was this just because you had a look around in the file system, or did you suffer from low disk space?
>>>>> 
>>>>> Just looking…
>>>> 
>>>> :)
>>>> 
>>>>> 
>>>>>> 
>>>>>>> And among those were lots of old unused openvpn graphs.  I think I had 8-10 old openvpn graphs that were more than 1 year old.
>>>>>>> 
>>>>>>> I had started to write a wiki page for users about finding and deleting old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs.  And then the wiki grew to ALL old graphs.  
>>>>>>> 
>>>>>>> Below is a snipped version of my original `find`.   I think there were ~40 total old obsolete RRD files.
>>>>>>> 
>>>>>>> ```
>>>>>>> [root(a)ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls
>>>>>>>    1838    304 -rw-r--r--   1 root     root       307712 Nov 17  2019 /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd
>>>>>>>    1794    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd
>>>>>>>    1821    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd
>>>>>>> . . .
>>>>>>>    1831    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd
>>>>>>>    1888    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd
>>>>>>>    1785    152 -rw-r--r--   1 root     root       154888 Aug 12  2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd
>>>>>>> . . .
>>>>>>>    1791    152 -rw-r--r--   1 root     root       154888 Aug 12  2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd
>>>>>>>    1880    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd
>>>>>>>    1875    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd
>>>>>>>    1874    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd
>>>>>>>    1781    152 -rw-r--r--   1 root     root       154888 Jan 13  2020 /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd
>>>>>>>    1937     56 -rw-r--r--   1 nobody   nobody      56800 Jul 18  2021 /var/log/rrd/iptraffic/192.168.60.105.rrd
>>>>>>>    1938     56 -rw-r--r--   1 nobody   nobody      56800 Jul 18  2021 /var/log/rrd/iptraffic/192.168.60.147.rrd
>>>>>>>    1935     56 -rw-r--r--   1 nobody   nobody      56800 Jul 18  2021 /var/log/rrd/iptraffic/192.168.60.194.rrd
>>>>>>> [root(a)ipfire ~] # 
>>>>>>> ```
>>>>>>> 
>>>>>>> As you can see some of the files were left over from 2017 and they did not get cleaned out during hardware changes.  I obviously did not follow the migrations wiki!
>>>>>> 
>>>>>> Well, I suppose if it doesn’t happen automatically, we cannot rely on this :) I never follow this either.
>>>>>> 
>>>>>>> The wiki page was to be instructions for users to `find` the files and then to delete the files via the command line.  While is was writing the wiki I remembered a Dev Mailing list message about sending users to the shell partially defeats the purpose of IPFire (bad paraphrase!).  So instead of writing a wiki I wrote it a weekly fcronjob to the help everyone.
>>>>>> 
>>>>>>> More than you ever wanted to know!  Hope this helps!
>>>>>> 
>>>>>> Yes it does, but in the end my concern remains:
>>>>>> 
>>>>>> Is it not possible that an OpenVPN connection is only being used once a year or something like that? Would we then not wipe any data? Wouldn’t those databases come back again?
>>>>> 
>>>>> I will test by wiping the current data and then connecting via OpenVPN to see if the databases and graphs recover.
>>>> 
>>>> I am quite confident that should work just fine.
>> 
>> I did a quick test and the folder and RRD file are both recreated.
>> 
>> Cool!
>> 
>>>> 
>>>>>> 
>>>>>> I suppose a database that has not been touched in over a year would not be carrying any information because that would have been rotated out of it (because that is how RRD databases work). So maybe that can go. But I would argue only then. So we would have to check what the longest time is those databases retain any information.
>>>>>> 
>>>>> 
>>>>> Who is the RRD guru for IPFire?  
>>>>> 
>>>>> I think there is an RRD dump command.  I will research.
>>>> 
>>>>  rrdtool dump /var/log/rrd/collectd/localhost/entropy/entropy.rrd
>>>> 
>>>> That will show you a looooong list with all the data points.
>>> 
>>> Looks like it lasts a year!
>>> 
>>> Beginning of a section of round robin:
>>> 
>>> ```
>>> 		</database>
>>> 	</rra>
>>> 	<rra>
>>> 		<cf>AVERAGE</cf>
>>> 		<pdp_per_row>878</pdp_per_row> <!-- 26340 seconds -->
>>> 
>>> 		<params>
>>> 		<xff>1.0000000000e-01</xff>
>>> 		</params>
>>> 		<cdp_prep>
>>> 			<ds>
>>> 			<primary_value>2.5600000000e+02</primary_value>
>>> 			<secondary_value>2.5600000000e+02</secondary_value>
>>> 			<value>2.0326400000e+05</value>
>>> 			<unknown_datapoints>0</unknown_datapoints>
>>> 			</ds>
>>> 		</cdp_prep>
>>> 		<database>
>>> 			<!-- 2021-08-24 09:04:00 CDT / 1629813840 --> <row><v>3.3458068337e+03</v></row>
>>> 			<!-- 2021-08-24 16:23:00 CDT / 1629840180 --> <row><v>3.3946542141e+03</v></row>
>>> 			<!-- 2021-08-24 23:42:00 CDT / 1629866520 --> <row><v>3.3813018223e+03</v></row>
>>> 			<!-- 2021-08-25 07:01:00 CDT / 1629892860 --> <row><v>3.3501125285e+03</v></row>
>>> 
>>> ```
>>> 
>>> End of the round robin:
>>> 
>>> ```
>>> 			<!-- 2022-08-23 09:10:00 CDT / 1661263800 --> <row><v>2.5600000000e+02</v></row>
>>> 			<!-- 2022-08-23 16:29:00 CDT / 1661290140 --> <row><v>2.5600000000e+02</v></row>
>>> 			<!-- 2022-08-23 23:48:00 CDT / 1661316480 --> <row><v>2.5600000000e+02</v></row>
>>> 			<!-- 2022-08-24 07:07:00 CDT / 1661342820 --> <row><v>2.5600000000e+02</v></row>
>>> 			<!-- 2022-08-24 14:26:00 CDT / 1661369160 --> <row><v>2.5600000000e+02</v></row>
>>> 			<!-- 2022-08-24 21:45:00 CDT / 1661395500 --> <row><v>2.5600000000e+02</v></row>
>>> 			<!-- 2022-08-25 05:04:00 CDT / 1661421840 --> <row><v>2.5600000000e+02</v></row>
>>> 		</database>
>>> 	</rra>
>>> 	<rra>
>>> 		<cf>MIN</cf>
>>> 		<pdp_per_row>878</pdp_per_row> <!-- 26340 seconds -->
>>> 
>>> ```
>>> 
>>>> 
>>>>>> Personally, I would redesign this whole thing a little. I would make them store a lot more data and I would even look for more metrics to collect. So, the extra space for a RRD database wouldn’t really bother me, even if the whole directory is going to use up gigabytes. If that is a problem for other users, I would like to know.
>>>>> 
>>>>> OK on my side!  I am at 48% (Use%) for a 16 GB drive.
>>>> 
>>>> Yes, 16GB is perfectly fine.
>>>> 
>>>>> I’d like to see a periodic bandwidth test (speedtest).  Once in awhile I drop to near 0 Mbps bandwidth.  Then it is time to restart the cable gateway and maybe the local network.
>>>> 
>>>> Yeah, that would be really nice to have :) I am just not sure how much traffic all IPFire devices like that would create.
>>>> 
>>>> Best,
>>>> -Michael
>>>> 
>>>>> 
>>>>> 
>>>>> Jon
>>>>> 
>>>>>> 
>>>>>> Best,
>>>>>> -Michael
>>>>>> 
>>>>>>> Jon
>>>>>>> 
>>>>>>> 
>>>>>>>> On Aug 25, 2022, at 5:48 AM, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>>>>> 
>>>>>>>> Hello Jon,
>>>>>>>> 
>>>>>>>> Thanks for the patch.
>>>>>>>> 
>>>>>>>> Sorry if this sounds like a stupid question, but what is the rationale for this?
>>>>>>>> 
>>>>>>>> -Michael
>>>>>>>> 
>>>>>>>>> On 25 Aug 2022, at 01:31, Jon Murphy <jon.murphy(a)ipfire.org> wrote:
>>>>>>>>> 
>>>>>>>>> - Created (mostly) for old openvpn graphs
>>>>>>>>> - RRD removed when no graph modification for +365 days
>>>>>>>>> - chosen since graph max out is 365 days
>>>>>>>>> - fcron job runs once per week
>>>>>>>>> - chosen since this is just a cleanup and it doesnt need to run everyday
>>>>>>>>> Note: logging can be added if needed.
>>>>>>>>> 
>>>>>>>>> Signed-off-by: Jon Murphy <jon.murphy(a)ipfire.org>
>>>>>>>>> ---
>>>>>>>>> config/cron/crontab | 3 +++
>>>>>>>>> 1 file changed, 3 insertions(+)
>>>>>>>>> 
>>>>>>>>> diff --git a/config/cron/crontab b/config/cron/crontab
>>>>>>>>> index b727531fc..7088e0b75 100644
>>>>>>>>> --- a/config/cron/crontab
>>>>>>>>> +++ b/config/cron/crontab
>>>>>>>>> @@ -80,5 +80,8 @@ HOME=/
>>>>>>>>> # Cleanup the mail spool directory
>>>>>>>>> %weekly * * /usr/sbin/dma-cleanup-spool
>>>>>>>>> 
>>>>>>>>> +# Cleanup the collectd RRD (graphs)
>>>>>>>>> +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete
>>>>>>>>> +
>>>>>>>>> # Update DNS trust anchor
>>>>>>>>> %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem
>>>>>>>>> -- 
>>>>>>>>> 2.30.2
>>> 
>> 
>