From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: [PATCH] crontab: add periodic cleanup the collectd RRD (graphs) Date: Tue, 20 Sep 2022 09:58:38 +0200 Message-ID: In-Reply-To: <39BEBD7B-18C4-4CEB-84DD-CDFC5227866A@gmail.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6648559813054360788==" List-Id: --===============6648559813054360788== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable I think this is fine for me: Reviewed-by: Michael Tremer Thanks for the nudge. -Michael > On 20 Sep 2022, at 03:28, Jon Murphy wrote: >=20 > Bump / nudge... Is this OK? >=20 >=20 > Jon >=20 >=20 >> On Aug 25, 2022, at 4:34 PM, Jon Murphy wrote: >>=20 >> Michael, >>=20 >> Last item below=E2=80=A6 >>=20 >> If OK please add your "Acked-by:" or "Reviewed-by:". >>=20 >> Jon >>=20 >>> On Aug 25, 2022, at 11:47 AM, Jon Murphy wrote: >>>=20 >>> Michael, >>>=20 >>>> On Aug 25, 2022, at 11:32 AM, Michael Tremer wrote: >>>>=20 >>>> Hello, >>>>=20 >>>>> On 25 Aug 2022, at 17:17, Jon Murphy wrote: >>>>>=20 >>>>> Michael, >>>>>=20 >>>>>=20 >>>>>> On Aug 25, 2022, at 10:44 AM, Michael Tremer wrote: >>>>>>=20 >>>>>> Hello Jon, >>>>>>=20 >>>>>>> On 25 Aug 2022, at 15:40, Jon Murphy wrote: >>>>>>>=20 >>>>>>> Hey Michael, >>>>>>>=20 >>>>>>> Over the years I=E2=80=99ve used four different IPFire boxes. And du= ring a "late-summer cleaning" I found lots of old RRD files. I ran this comm= and:=20 >>>>>>> `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls` >>>>>>=20 >>>>>> Was this just because you had a look around in the file system, or did= you suffer from low disk space? >>>>>=20 >>>>> Just looking=E2=80=A6 >>>>=20 >>>> :) >>>>=20 >>>>>=20 >>>>>>=20 >>>>>>> And among those were lots of old unused openvpn graphs. I think I ha= d 8-10 old openvpn graphs that were more than 1 year old. >>>>>>>=20 >>>>>>> I had started to write a wiki page for users about finding and deleti= ng old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs. And then the wiki g= rew to ALL old graphs. =20 >>>>>>>=20 >>>>>>> Below is a snipped version of my original `find`. I think there wer= e ~40 total old obsolete RRD files. >>>>>>>=20 >>>>>>> ``` >>>>>>> [root(a)ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rr= d' -ls >>>>>>> 1838 304 -rw-r--r-- 1 root root 307712 Nov 17 2019= /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd >>>>>>> 1794 152 -rw-r--r-- 1 root root 154888 Jul 18 2021= /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1= .rrd >>>>>>> 1821 152 -rw-r--r-- 1 root root 154888 Jul 18 2021= /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd >>>>>>> . . . >>>>>>> 1831 152 -rw-r--r-- 1 root root 154888 Jul 18 2021= /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd >>>>>>> 1888 152 -rw-r--r-- 1 root root 154888 Jul 18 2021= /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1= .rrd >>>>>>> 1785 152 -rw-r--r-- 1 root root 154888 Aug 12 2017= /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp= 2.rrd >>>>>>> . . . >>>>>>> 1791 152 -rw-r--r-- 1 root root 154888 Aug 12 2017= /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd >>>>>>> 1880 152 -rw-r--r-- 1 root root 154888 Jul 18 2021= /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperatur= e.rrd >>>>>>> 1875 152 -rw-r--r-- 1 root root 154888 Jul 18 2021= /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.= rrd >>>>>>> 1874 152 -rw-r--r-- 1 root root 154888 Jul 18 2021= /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.= rrd >>>>>>> 1781 152 -rw-r--r-- 1 root root 154888 Jan 13 2020= /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperatur= e.rrd >>>>>>> 1937 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021= /var/log/rrd/iptraffic/192.168.60.105.rrd >>>>>>> 1938 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021= /var/log/rrd/iptraffic/192.168.60.147.rrd >>>>>>> 1935 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021= /var/log/rrd/iptraffic/192.168.60.194.rrd >>>>>>> [root(a)ipfire ~] #=20 >>>>>>> ``` >>>>>>>=20 >>>>>>> As you can see some of the files were left over from 2017 and they di= d not get cleaned out during hardware changes. I obviously did not follow th= e migrations wiki! >>>>>>=20 >>>>>> Well, I suppose if it doesn=E2=80=99t happen automatically, we cannot = rely on this :) I never follow this either. >>>>>>=20 >>>>>>> The wiki page was to be instructions for users to `find` the files an= d then to delete the files via the command line. While is was writing the wi= ki I remembered a Dev Mailing list message about sending users to the shell p= artially defeats the purpose of IPFire (bad paraphrase!). So instead of writ= ing a wiki I wrote it a weekly fcronjob to the help everyone. >>>>>>=20 >>>>>>> More than you ever wanted to know! Hope this helps! >>>>>>=20 >>>>>> Yes it does, but in the end my concern remains: >>>>>>=20 >>>>>> Is it not possible that an OpenVPN connection is only being used once = a year or something like that? Would we then not wipe any data? Wouldn=E2=80= =99t those databases come back again? >>>>>=20 >>>>> I will test by wiping the current data and then connecting via OpenVPN = to see if the databases and graphs recover. >>>>=20 >>>> I am quite confident that should work just fine. >>=20 >> I did a quick test and the folder and RRD file are both recreated. >>=20 >> Cool! >>=20 >>>>=20 >>>>>>=20 >>>>>> I suppose a database that has not been touched in over a year would no= t be carrying any information because that would have been rotated out of it = (because that is how RRD databases work). So maybe that can go. But I would a= rgue only then. So we would have to check what the longest time is those data= bases retain any information. >>>>>>=20 >>>>>=20 >>>>> Who is the RRD guru for IPFire? =20 >>>>>=20 >>>>> I think there is an RRD dump command. I will research. >>>>=20 >>>> rrdtool dump /var/log/rrd/collectd/localhost/entropy/entropy.rrd >>>>=20 >>>> That will show you a looooong list with all the data points. >>>=20 >>> Looks like it lasts a year! >>>=20 >>> Beginning of a section of round robin: >>>=20 >>> ``` >>> >>> >>> >>> AVERAGE >>> 878 >>>=20 >>> >>> 1.0000000000e-01 >>> >>> >>> >>> 2.5600000000e+02 >>> 2.5600000000e+02 >>> 2.0326400000e+05 >>> 0 >>> >>> >>> >>> 3.3458068337e+03= >>> 3.3946542141e+03= >>> 3.3813018223e+03= >>> 3.3501125285e+03= >>>=20 >>> ``` >>>=20 >>> End of the round robin: >>>=20 >>> ``` >>> 2.5600000000e+02= >>> 2.5600000000e+02= >>> 2.5600000000e+02= >>> 2.5600000000e+02= >>> 2.5600000000e+02= >>> 2.5600000000e+02= >>> 2.5600000000e+02= >>> >>> >>> >>> MIN >>> 878 >>>=20 >>> ``` >>>=20 >>>>=20 >>>>>> Personally, I would redesign this whole thing a little. I would make t= hem store a lot more data and I would even look for more metrics to collect. = So, the extra space for a RRD database wouldn=E2=80=99t really bother me, eve= n if the whole directory is going to use up gigabytes. If that is a problem f= or other users, I would like to know. >>>>>=20 >>>>> OK on my side! I am at 48% (Use%) for a 16 GB drive. >>>>=20 >>>> Yes, 16GB is perfectly fine. >>>>=20 >>>>> I=E2=80=99d like to see a periodic bandwidth test (speedtest). Once in= awhile I drop to near 0 Mbps bandwidth. Then it is time to restart the cabl= e gateway and maybe the local network. >>>>=20 >>>> Yeah, that would be really nice to have :) I am just not sure how much t= raffic all IPFire devices like that would create. >>>>=20 >>>> Best, >>>> -Michael >>>>=20 >>>>>=20 >>>>>=20 >>>>> Jon >>>>>=20 >>>>>>=20 >>>>>> Best, >>>>>> -Michael >>>>>>=20 >>>>>>> Jon >>>>>>>=20 >>>>>>>=20 >>>>>>>> On Aug 25, 2022, at 5:48 AM, Michael Tremer wrote: >>>>>>>>=20 >>>>>>>> Hello Jon, >>>>>>>>=20 >>>>>>>> Thanks for the patch. >>>>>>>>=20 >>>>>>>> Sorry if this sounds like a stupid question, but what is the rationa= le for this? >>>>>>>>=20 >>>>>>>> -Michael >>>>>>>>=20 >>>>>>>>> On 25 Aug 2022, at 01:31, Jon Murphy wrot= e: >>>>>>>>>=20 >>>>>>>>> - Created (mostly) for old openvpn graphs >>>>>>>>> - RRD removed when no graph modification for +365 days >>>>>>>>> - chosen since graph max out is 365 days >>>>>>>>> - fcron job runs once per week >>>>>>>>> - chosen since this is just a cleanup and it doesnt need to run eve= ryday >>>>>>>>> Note: logging can be added if needed. >>>>>>>>>=20 >>>>>>>>> Signed-off-by: Jon Murphy >>>>>>>>> --- >>>>>>>>> config/cron/crontab | 3 +++ >>>>>>>>> 1 file changed, 3 insertions(+) >>>>>>>>>=20 >>>>>>>>> diff --git a/config/cron/crontab b/config/cron/crontab >>>>>>>>> index b727531fc..7088e0b75 100644 >>>>>>>>> --- a/config/cron/crontab >>>>>>>>> +++ b/config/cron/crontab >>>>>>>>> @@ -80,5 +80,8 @@ HOME=3D/ >>>>>>>>> # Cleanup the mail spool directory >>>>>>>>> %weekly * * /usr/sbin/dma-cleanup-spool >>>>>>>>>=20 >>>>>>>>> +# Cleanup the collectd RRD (graphs) >>>>>>>>> +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rr= d' -delete -o -type d -empty -delete >>>>>>>>> + >>>>>>>>> # Update DNS trust anchor >>>>>>>>> %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/l= ib/unbound/root.key -c /etc/unbound/icannbundle.pem >>>>>>>>> --=20 >>>>>>>>> 2.30.2 >>>=20 >>=20 >=20 --===============6648559813054360788==--