Re: [PATCH] crontab: add periodic cleanup the collectd RRD (graphs)

Re: [PATCH] crontab: add periodic cleanup the collectd RRD (graphs)

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 10517 bytes --]

I think this is fine for me:

Reviewed-by: Michael Tremer <michael.tremer(a)ipfire.org>

Thanks for the nudge.

-Michael

> On 20 Sep 2022, at 03:28, Jon Murphy <jcmurphy26(a)gmail.com> wrote:
> 
> Bump / nudge...    Is this OK?
> 
> 
> Jon
> 
> 
>> On Aug 25, 2022, at 4:34 PM, Jon Murphy <jcmurphy26(a)gmail.com> wrote:
>> 
>> Michael,
>> 
>> Last item below…
>> 
>> If OK please add your "Acked-by:" or "Reviewed-by:".
>> 
>> Jon
>> 
>>> On Aug 25, 2022, at 11:47 AM, Jon Murphy <jcmurphy26(a)gmail.com> wrote:
>>> 
>>> Michael,
>>> 
>>>> On Aug 25, 2022, at 11:32 AM, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>> 
>>>> Hello,
>>>> 
>>>>> On 25 Aug 2022, at 17:17, Jon Murphy <jcmurphy26(a)gmail.com> wrote:
>>>>> 
>>>>> Michael,
>>>>> 
>>>>> 
>>>>>> On Aug 25, 2022, at 10:44 AM, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>>> 
>>>>>> Hello Jon,
>>>>>> 
>>>>>>> On 25 Aug 2022, at 15:40, Jon Murphy <jcmurphy26(a)gmail.com> wrote:
>>>>>>> 
>>>>>>> Hey Michael,
>>>>>>> 
>>>>>>> Over the years I’ve used four different IPFire boxes.  And during a "late-summer cleaning" I found lots of old RRD files.  I ran this command: 
>>>>>>> `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls`
>>>>>> 
>>>>>> Was this just because you had a look around in the file system, or did you suffer from low disk space?
>>>>> 
>>>>> Just looking…
>>>> 
>>>> :)
>>>> 
>>>>> 
>>>>>> 
>>>>>>> And among those were lots of old unused openvpn graphs.  I think I had 8-10 old openvpn graphs that were more than 1 year old.
>>>>>>> 
>>>>>>> I had started to write a wiki page for users about finding and deleting old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs.  And then the wiki grew to ALL old graphs.  
>>>>>>> 
>>>>>>> Below is a snipped version of my original `find`.   I think there were ~40 total old obsolete RRD files.
>>>>>>> 
>>>>>>> ```
>>>>>>> [root(a)ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls
>>>>>>>    1838    304 -rw-r--r--   1 root     root       307712 Nov 17  2019 /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd
>>>>>>>    1794    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd
>>>>>>>    1821    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd
>>>>>>> . . .
>>>>>>>    1831    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd
>>>>>>>    1888    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd
>>>>>>>    1785    152 -rw-r--r--   1 root     root       154888 Aug 12  2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd
>>>>>>> . . .
>>>>>>>    1791    152 -rw-r--r--   1 root     root       154888 Aug 12  2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd
>>>>>>>    1880    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd
>>>>>>>    1875    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd
>>>>>>>    1874    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd
>>>>>>>    1781    152 -rw-r--r--   1 root     root       154888 Jan 13  2020 /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd
>>>>>>>    1937     56 -rw-r--r--   1 nobody   nobody      56800 Jul 18  2021 /var/log/rrd/iptraffic/192.168.60.105.rrd
>>>>>>>    1938     56 -rw-r--r--   1 nobody   nobody      56800 Jul 18  2021 /var/log/rrd/iptraffic/192.168.60.147.rrd
>>>>>>>    1935     56 -rw-r--r--   1 nobody   nobody      56800 Jul 18  2021 /var/log/rrd/iptraffic/192.168.60.194.rrd
>>>>>>> [root(a)ipfire ~] # 
>>>>>>> ```
>>>>>>> 
>>>>>>> As you can see some of the files were left over from 2017 and they did not get cleaned out during hardware changes.  I obviously did not follow the migrations wiki!
>>>>>> 
>>>>>> Well, I suppose if it doesn’t happen automatically, we cannot rely on this :) I never follow this either.
>>>>>> 
>>>>>>> The wiki page was to be instructions for users to `find` the files and then to delete the files via the command line.  While is was writing the wiki I remembered a Dev Mailing list message about sending users to the shell partially defeats the purpose of IPFire (bad paraphrase!).  So instead of writing a wiki I wrote it a weekly fcronjob to the help everyone.
>>>>>> 
>>>>>>> More than you ever wanted to know!  Hope this helps!
>>>>>> 
>>>>>> Yes it does, but in the end my concern remains:
>>>>>> 
>>>>>> Is it not possible that an OpenVPN connection is only being used once a year or something like that? Would we then not wipe any data? Wouldn’t those databases come back again?
>>>>> 
>>>>> I will test by wiping the current data and then connecting via OpenVPN to see if the databases and graphs recover.
>>>> 
>>>> I am quite confident that should work just fine.
>> 
>> I did a quick test and the folder and RRD file are both recreated.
>> 
>> Cool!
>> 
>>>> 
>>>>>> 
>>>>>> I suppose a database that has not been touched in over a year would not be carrying any information because that would have been rotated out of it (because that is how RRD databases work). So maybe that can go. But I would argue only then. So we would have to check what the longest time is those databases retain any information.
>>>>>> 
>>>>> 
>>>>> Who is the RRD guru for IPFire?  
>>>>> 
>>>>> I think there is an RRD dump command.  I will research.
>>>> 
>>>>  rrdtool dump /var/log/rrd/collectd/localhost/entropy/entropy.rrd
>>>> 
>>>> That will show you a looooong list with all the data points.
>>> 
>>> Looks like it lasts a year!
>>> 
>>> Beginning of a section of round robin:
>>> 
>>> ```
>>> 		</database>
>>> 	</rra>
>>> 	<rra>
>>> 		<cf>AVERAGE</cf>
>>> 		<pdp_per_row>878</pdp_per_row> <!-- 26340 seconds -->
>>> 
>>> 		<params>
>>> 		<xff>1.0000000000e-01</xff>
>>> 		</params>
>>> 		<cdp_prep>
>>> 			<ds>
>>> 			<primary_value>2.5600000000e+02</primary_value>
>>> 			<secondary_value>2.5600000000e+02</secondary_value>
>>> 			<value>2.0326400000e+05</value>
>>> 			<unknown_datapoints>0</unknown_datapoints>
>>> 			</ds>
>>> 		</cdp_prep>
>>> 		<database>
>>> 			<!-- 2021-08-24 09:04:00 CDT / 1629813840 --> <row><v>3.3458068337e+03</v></row>
>>> 			<!-- 2021-08-24 16:23:00 CDT / 1629840180 --> <row><v>3.3946542141e+03</v></row>
>>> 			<!-- 2021-08-24 23:42:00 CDT / 1629866520 --> <row><v>3.3813018223e+03</v></row>
>>> 			<!-- 2021-08-25 07:01:00 CDT / 1629892860 --> <row><v>3.3501125285e+03</v></row>
>>> 
>>> ```
>>> 
>>> End of the round robin:
>>> 
>>> ```
>>> 			<!-- 2022-08-23 09:10:00 CDT / 1661263800 --> <row><v>2.5600000000e+02</v></row>
>>> 			<!-- 2022-08-23 16:29:00 CDT / 1661290140 --> <row><v>2.5600000000e+02</v></row>
>>> 			<!-- 2022-08-23 23:48:00 CDT / 1661316480 --> <row><v>2.5600000000e+02</v></row>
>>> 			<!-- 2022-08-24 07:07:00 CDT / 1661342820 --> <row><v>2.5600000000e+02</v></row>
>>> 			<!-- 2022-08-24 14:26:00 CDT / 1661369160 --> <row><v>2.5600000000e+02</v></row>
>>> 			<!-- 2022-08-24 21:45:00 CDT / 1661395500 --> <row><v>2.5600000000e+02</v></row>
>>> 			<!-- 2022-08-25 05:04:00 CDT / 1661421840 --> <row><v>2.5600000000e+02</v></row>
>>> 		</database>
>>> 	</rra>
>>> 	<rra>
>>> 		<cf>MIN</cf>
>>> 		<pdp_per_row>878</pdp_per_row> <!-- 26340 seconds -->
>>> 
>>> ```
>>> 
>>>> 
>>>>>> Personally, I would redesign this whole thing a little. I would make them store a lot more data and I would even look for more metrics to collect. So, the extra space for a RRD database wouldn’t really bother me, even if the whole directory is going to use up gigabytes. If that is a problem for other users, I would like to know.
>>>>> 
>>>>> OK on my side!  I am at 48% (Use%) for a 16 GB drive.
>>>> 
>>>> Yes, 16GB is perfectly fine.
>>>> 
>>>>> I’d like to see a periodic bandwidth test (speedtest).  Once in awhile I drop to near 0 Mbps bandwidth.  Then it is time to restart the cable gateway and maybe the local network.
>>>> 
>>>> Yeah, that would be really nice to have :) I am just not sure how much traffic all IPFire devices like that would create.
>>>> 
>>>> Best,
>>>> -Michael
>>>> 
>>>>> 
>>>>> 
>>>>> Jon
>>>>> 
>>>>>> 
>>>>>> Best,
>>>>>> -Michael
>>>>>> 
>>>>>>> Jon
>>>>>>> 
>>>>>>> 
>>>>>>>> On Aug 25, 2022, at 5:48 AM, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>>>>>> 
>>>>>>>> Hello Jon,
>>>>>>>> 
>>>>>>>> Thanks for the patch.
>>>>>>>> 
>>>>>>>> Sorry if this sounds like a stupid question, but what is the rationale for this?
>>>>>>>> 
>>>>>>>> -Michael
>>>>>>>> 
>>>>>>>>> On 25 Aug 2022, at 01:31, Jon Murphy <jon.murphy(a)ipfire.org> wrote:
>>>>>>>>> 
>>>>>>>>> - Created (mostly) for old openvpn graphs
>>>>>>>>> - RRD removed when no graph modification for +365 days
>>>>>>>>> - chosen since graph max out is 365 days
>>>>>>>>> - fcron job runs once per week
>>>>>>>>> - chosen since this is just a cleanup and it doesnt need to run everyday
>>>>>>>>> Note: logging can be added if needed.
>>>>>>>>> 
>>>>>>>>> Signed-off-by: Jon Murphy <jon.murphy(a)ipfire.org>
>>>>>>>>> ---
>>>>>>>>> config/cron/crontab | 3 +++
>>>>>>>>> 1 file changed, 3 insertions(+)
>>>>>>>>> 
>>>>>>>>> diff --git a/config/cron/crontab b/config/cron/crontab
>>>>>>>>> index b727531fc..7088e0b75 100644
>>>>>>>>> --- a/config/cron/crontab
>>>>>>>>> +++ b/config/cron/crontab
>>>>>>>>> @@ -80,5 +80,8 @@ HOME=/
>>>>>>>>> # Cleanup the mail spool directory
>>>>>>>>> %weekly * * /usr/sbin/dma-cleanup-spool
>>>>>>>>> 
>>>>>>>>> +# Cleanup the collectd RRD (graphs)
>>>>>>>>> +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete
>>>>>>>>> +
>>>>>>>>> # Update DNS trust anchor
>>>>>>>>> %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem
>>>>>>>>> -- 
>>>>>>>>> 2.30.2
>>> 
>> 
> 

Re: [PATCH] crontab: add periodic cleanup the collectd RRD (graphs)

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 7373 bytes --]

Hello,

> On 25 Aug 2022, at 17:17, Jon Murphy <jcmurphy26(a)gmail.com> wrote:
> 
> Michael,
> 
> 
>> On Aug 25, 2022, at 10:44 AM, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>> 
>> Hello Jon,
>> 
>>> On 25 Aug 2022, at 15:40, Jon Murphy <jcmurphy26(a)gmail.com> wrote:
>>> 
>>> Hey Michael,
>>> 
>>> Over the years I’ve used four different IPFire boxes.  And during a "late-summer cleaning" I found lots of old RRD files.  I ran this command: 
>>> `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls`
>> 
>> Was this just because you had a look around in the file system, or did you suffer from low disk space?
> 
> Just looking…

:)

> 
>> 
>>> And among those were lots of old unused openvpn graphs.  I think I had 8-10 old openvpn graphs that were more than 1 year old.
>>> 
>>> I had started to write a wiki page for users about finding and deleting old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs.  And then the wiki grew to ALL old graphs.  
>>> 
>>> Below is a snipped version of my original `find`.   I think there were ~40 total old obsolete RRD files.
>>> 
>>> ```
>>> [root(a)ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls
>>>     1838    304 -rw-r--r--   1 root     root       307712 Nov 17  2019 /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd
>>>     1794    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd
>>>     1821    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd
>>> . . .
>>>     1831    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd
>>>     1888    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd
>>>     1785    152 -rw-r--r--   1 root     root       154888 Aug 12  2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd
>>> . . .
>>>     1791    152 -rw-r--r--   1 root     root       154888 Aug 12  2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd
>>>     1880    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd
>>>     1875    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd
>>>     1874    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd
>>>     1781    152 -rw-r--r--   1 root     root       154888 Jan 13  2020 /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd
>>>     1937     56 -rw-r--r--   1 nobody   nobody      56800 Jul 18  2021 /var/log/rrd/iptraffic/192.168.60.105.rrd
>>>     1938     56 -rw-r--r--   1 nobody   nobody      56800 Jul 18  2021 /var/log/rrd/iptraffic/192.168.60.147.rrd
>>>     1935     56 -rw-r--r--   1 nobody   nobody      56800 Jul 18  2021 /var/log/rrd/iptraffic/192.168.60.194.rrd
>>> [root(a)ipfire ~] # 
>>> ```
>>> 
>>> As you can see some of the files were left over from 2017 and they did not get cleaned out during hardware changes.  I obviously did not follow the migrations wiki!
>> 
>> Well, I suppose if it doesn’t happen automatically, we cannot rely on this :) I never follow this either.
>> 
>>> The wiki page was to be instructions for users to `find` the files and then to delete the files via the command line.  While is was writing the wiki I remembered a Dev Mailing list message about sending users to the shell partially defeats the purpose of IPFire (bad paraphrase!).  So instead of writing a wiki I wrote it a weekly fcronjob to the help everyone.
>> 
>>> More than you ever wanted to know!  Hope this helps!
>> 
>> Yes it does, but in the end my concern remains:
>> 
>> Is it not possible that an OpenVPN connection is only being used once a year or something like that? Would we then not wipe any data? Wouldn’t those databases come back again?
> 
> I will test by wiping the current data and then connecting via OpenVPN to see if the databases and graphs recover.

I am quite confident that should work just fine.

>> 
>> I suppose a database that has not been touched in over a year would not be carrying any information because that would have been rotated out of it (because that is how RRD databases work). So maybe that can go. But I would argue only then. So we would have to check what the longest time is those databases retain any information.
>> 
> 
> Who is the RRD guru for IPFire?  
> 
> I think there is an RRD dump command.  I will research.

  rrdtool dump /var/log/rrd/collectd/localhost/entropy/entropy.rrd

That will show you a looooong list with all the data points.

>> Personally, I would redesign this whole thing a little. I would make them store a lot more data and I would even look for more metrics to collect. So, the extra space for a RRD database wouldn’t really bother me, even if the whole directory is going to use up gigabytes. If that is a problem for other users, I would like to know.
> 
> OK on my side!  I am at 48% (Use%) for a 16 GB drive.

Yes, 16GB is perfectly fine.

> I’d like to see a periodic bandwidth test (speedtest).  Once in awhile I drop to near 0 Mbps bandwidth.  Then it is time to restart the cable gateway and maybe the local network.

Yeah, that would be really nice to have :) I am just not sure how much traffic all IPFire devices like that would create.

Best,
-Michael

> 
> 
> Jon
> 
>> 
>> Best,
>> -Michael
>> 
>>> Jon
>>> 
>>> 
>>>> On Aug 25, 2022, at 5:48 AM, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>>>> 
>>>> Hello Jon,
>>>> 
>>>> Thanks for the patch.
>>>> 
>>>> Sorry if this sounds like a stupid question, but what is the rationale for this?
>>>> 
>>>> -Michael
>>>> 
>>>>> On 25 Aug 2022, at 01:31, Jon Murphy <jon.murphy(a)ipfire.org> wrote:
>>>>> 
>>>>> - Created (mostly) for old openvpn graphs
>>>>> - RRD removed when no graph modification for +365 days
>>>>> - chosen since graph max out is 365 days
>>>>> - fcron job runs once per week
>>>>> - chosen since this is just a cleanup and it doesnt need to run everyday
>>>>> Note: logging can be added if needed.
>>>>> 
>>>>> Signed-off-by: Jon Murphy <jon.murphy(a)ipfire.org>
>>>>> ---
>>>>> config/cron/crontab | 3 +++
>>>>> 1 file changed, 3 insertions(+)
>>>>> 
>>>>> diff --git a/config/cron/crontab b/config/cron/crontab
>>>>> index b727531fc..7088e0b75 100644
>>>>> --- a/config/cron/crontab
>>>>> +++ b/config/cron/crontab
>>>>> @@ -80,5 +80,8 @@ HOME=/
>>>>> # Cleanup the mail spool directory
>>>>> %weekly * * /usr/sbin/dma-cleanup-spool
>>>>> 
>>>>> +# Cleanup the collectd RRD (graphs)
>>>>> +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete
>>>>> +
>>>>> # Update DNS trust anchor
>>>>> %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem
>>>>> -- 
>>>>> 2.30.2

Re: [PATCH] crontab: add periodic cleanup the collectd RRD (graphs)

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 6091 bytes --]

Hello Jon,

> On 25 Aug 2022, at 15:40, Jon Murphy <jcmurphy26(a)gmail.com> wrote:
> 
> Hey Michael,
> 
> Over the years I’ve used four different IPFire boxes.  And during a "late-summer cleaning" I found lots of old RRD files.  I ran this command: 
> `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls`

Was this just because you had a look around in the file system, or did you suffer from low disk space?

> And among those were lots of old unused openvpn graphs.  I think I had 8-10 old openvpn graphs that were more than 1 year old.
> 
> I had started to write a wiki page for users about finding and deleting old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs.  And then the wiki grew to ALL old graphs.  
> 
> Below is a snipped version of my original `find`.   I think there were ~40 total old obsolete RRD files.
> 
> ```
> [root(a)ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls
>      1838    304 -rw-r--r--   1 root     root       307712 Nov 17  2019 /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd
>      1794    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd
>      1821    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd
> . . .
>      1831    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd
>      1888    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd
>      1785    152 -rw-r--r--   1 root     root       154888 Aug 12  2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd
> . . .
>      1791    152 -rw-r--r--   1 root     root       154888 Aug 12  2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd
>      1880    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd
>      1875    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd
>      1874    152 -rw-r--r--   1 root     root       154888 Jul 18  2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd
>      1781    152 -rw-r--r--   1 root     root       154888 Jan 13  2020 /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd
>      1937     56 -rw-r--r--   1 nobody   nobody      56800 Jul 18  2021 /var/log/rrd/iptraffic/192.168.60.105.rrd
>      1938     56 -rw-r--r--   1 nobody   nobody      56800 Jul 18  2021 /var/log/rrd/iptraffic/192.168.60.147.rrd
>      1935     56 -rw-r--r--   1 nobody   nobody      56800 Jul 18  2021 /var/log/rrd/iptraffic/192.168.60.194.rrd
> [root(a)ipfire ~] # 
> ```
> 
> As you can see some of the files were left over from 2017 and they did not get cleaned out during hardware changes.  I obviously did not follow the migrations wiki!

Well, I suppose if it doesn’t happen automatically, we cannot rely on this :) I never follow this either.

> The wiki page was to be instructions for users to `find` the files and then to delete the files via the command line.  While is was writing the wiki I remembered a Dev Mailing list message about sending users to the shell partially defeats the purpose of IPFire (bad paraphrase!).  So instead of writing a wiki I wrote it a weekly fcronjob to the help everyone.

> More than you ever wanted to know!  Hope this helps!

Yes it does, but in the end my concern remains:

Is it not possible that an OpenVPN connection is only being used once a year or something like that? Would we then not wipe any data? Wouldn’t those databases come back again?

I suppose a database that has not been touched in over a year would not be carrying any information because that would have been rotated out of it (because that is how RRD databases work). So maybe that can go. But I would argue only then. So we would have to check what the longest time is those databases retain any information.

Personally, I would redesign this whole thing a little. I would make them store a lot more data and I would even look for more metrics to collect. So, the extra space for a RRD database wouldn’t really bother me, even if the whole directory is going to use up gigabytes. If that is a problem for other users, I would like to know.

Best,
-Michael

> Jon
> 
> 
>> On Aug 25, 2022, at 5:48 AM, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
>> 
>> Hello Jon,
>> 
>> Thanks for the patch.
>> 
>> Sorry if this sounds like a stupid question, but what is the rationale for this?
>> 
>> -Michael
>> 
>>> On 25 Aug 2022, at 01:31, Jon Murphy <jon.murphy(a)ipfire.org> wrote:
>>> 
>>> - Created (mostly) for old openvpn graphs
>>> - RRD removed when no graph modification for +365 days
>>> - chosen since graph max out is 365 days
>>> - fcron job runs once per week
>>> - chosen since this is just a cleanup and it doesnt need to run everyday
>>> Note: logging can be added if needed.
>>> 
>>> Signed-off-by: Jon Murphy <jon.murphy(a)ipfire.org>
>>> ---
>>> config/cron/crontab | 3 +++
>>> 1 file changed, 3 insertions(+)
>>> 
>>> diff --git a/config/cron/crontab b/config/cron/crontab
>>> index b727531fc..7088e0b75 100644
>>> --- a/config/cron/crontab
>>> +++ b/config/cron/crontab
>>> @@ -80,5 +80,8 @@ HOME=/
>>> # Cleanup the mail spool directory
>>> %weekly * * /usr/sbin/dma-cleanup-spool
>>> 
>>> +# Cleanup the collectd RRD (graphs)
>>> +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete
>>> +
>>> # Update DNS trust anchor
>>> %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem
>>> -- 
>>> 2.30.2
>>> 
>> 
> 

Re: [PATCH] crontab: add periodic cleanup the collectd RRD (graphs)

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1240 bytes --]

Hello Jon,

Thanks for the patch.

Sorry if this sounds like a stupid question, but what is the rationale for this?

-Michael

> On 25 Aug 2022, at 01:31, Jon Murphy <jon.murphy(a)ipfire.org> wrote:
> 
> - Created (mostly) for old openvpn graphs
> - RRD removed when no graph modification for +365 days
>  - chosen since graph max out is 365 days
> - fcron job runs once per week
>  - chosen since this is just a cleanup and it doesnt need to run everyday
> Note: logging can be added if needed.
> 
> Signed-off-by: Jon Murphy <jon.murphy(a)ipfire.org>
> ---
> config/cron/crontab | 3 +++
> 1 file changed, 3 insertions(+)
> 
> diff --git a/config/cron/crontab b/config/cron/crontab
> index b727531fc..7088e0b75 100644
> --- a/config/cron/crontab
> +++ b/config/cron/crontab
> @@ -80,5 +80,8 @@ HOME=/
> # Cleanup the mail spool directory
> %weekly * * /usr/sbin/dma-cleanup-spool
> 
> +# Cleanup the collectd RRD (graphs)
> +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete
> +
> # Update DNS trust anchor
> %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem
> -- 
> 2.30.2
> 

Re: [PATCH] crontab: add periodic cleanup the collectd RRD (graphs)

[Bernhard Bitsch]

[-- Attachment #1: Type: text/plain, Size: 1122 bytes --]

Reviewed-by: Bernhard Bitsch <bbitsch(a)ipfire.org>

Am 25.08.2022 um 02:31 schrieb Jon Murphy:
> - Created (mostly) for old openvpn graphs
> - RRD removed when no graph modification for +365 days
>    - chosen since graph max out is 365 days
> - fcron job runs once per week
>    - chosen since this is just a cleanup and it doesnt need to run everyday
> Note: logging can be added if needed.
> 
> Signed-off-by: Jon Murphy <jon.murphy(a)ipfire.org>
> ---
>   config/cron/crontab | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/config/cron/crontab b/config/cron/crontab
> index b727531fc..7088e0b75 100644
> --- a/config/cron/crontab
> +++ b/config/cron/crontab
> @@ -80,5 +80,8 @@ HOME=/
>   # Cleanup the mail spool directory
>   %weekly * * /usr/sbin/dma-cleanup-spool
>   
> +# Cleanup the collectd RRD (graphs)
> +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete
> +
>   # Update DNS trust anchor
>   %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem

[PATCH] crontab: add periodic cleanup the collectd RRD (graphs)

[Jon Murphy]

[-- Attachment #1: Type: text/plain, Size: 979 bytes --]

- Created (mostly) for old openvpn graphs
- RRD removed when no graph modification for +365 days
  - chosen since graph max out is 365 days
- fcron job runs once per week
  - chosen since this is just a cleanup and it doesnt need to run everyday
Note: logging can be added if needed.

Signed-off-by: Jon Murphy <jon.murphy(a)ipfire.org>
---
 config/cron/crontab | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/config/cron/crontab b/config/cron/crontab
index b727531fc..7088e0b75 100644
--- a/config/cron/crontab
+++ b/config/cron/crontab
@@ -80,5 +80,8 @@ HOME=/
 # Cleanup the mail spool directory
 %weekly * * /usr/sbin/dma-cleanup-spool
 
+# Cleanup the collectd RRD (graphs)
+%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete
+
 # Update DNS trust anchor
 %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem
-- 
2.30.2