- Created (mostly) for old openvpn graphs - RRD removed when no graph modification for +365 days - chosen since graph max out is 365 days - fcron job runs once per week - chosen since this is just a cleanup and it doesnt need to run everyday Note: logging can be added if needed.
Signed-off-by: Jon Murphy jon.murphy@ipfire.org --- config/cron/crontab | 3 +++ 1 file changed, 3 insertions(+)
diff --git a/config/cron/crontab b/config/cron/crontab index b727531fc..7088e0b75 100644 --- a/config/cron/crontab +++ b/config/cron/crontab @@ -80,5 +80,8 @@ HOME=/ # Cleanup the mail spool directory %weekly * * /usr/sbin/dma-cleanup-spool
+# Cleanup the collectd RRD (graphs) +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete + # Update DNS trust anchor %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem
Reviewed-by: Bernhard Bitsch bbitsch@ipfire.org
Am 25.08.2022 um 02:31 schrieb Jon Murphy:
- Created (mostly) for old openvpn graphs
- RRD removed when no graph modification for +365 days
- chosen since graph max out is 365 days
- fcron job runs once per week
- chosen since this is just a cleanup and it doesnt need to run everyday
Note: logging can be added if needed.
Signed-off-by: Jon Murphy jon.murphy@ipfire.org
config/cron/crontab | 3 +++ 1 file changed, 3 insertions(+)
diff --git a/config/cron/crontab b/config/cron/crontab index b727531fc..7088e0b75 100644 --- a/config/cron/crontab +++ b/config/cron/crontab @@ -80,5 +80,8 @@ HOME=/ # Cleanup the mail spool directory %weekly * * /usr/sbin/dma-cleanup-spool
+# Cleanup the collectd RRD (graphs) +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete
- # Update DNS trust anchor %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem
Hello Jon,
Thanks for the patch.
Sorry if this sounds like a stupid question, but what is the rationale for this?
-Michael
On 25 Aug 2022, at 01:31, Jon Murphy jon.murphy@ipfire.org wrote:
- Created (mostly) for old openvpn graphs
- RRD removed when no graph modification for +365 days
- chosen since graph max out is 365 days
- fcron job runs once per week
- chosen since this is just a cleanup and it doesnt need to run everyday
Note: logging can be added if needed.
Signed-off-by: Jon Murphy jon.murphy@ipfire.org
config/cron/crontab | 3 +++ 1 file changed, 3 insertions(+)
diff --git a/config/cron/crontab b/config/cron/crontab index b727531fc..7088e0b75 100644 --- a/config/cron/crontab +++ b/config/cron/crontab @@ -80,5 +80,8 @@ HOME=/ # Cleanup the mail spool directory %weekly * * /usr/sbin/dma-cleanup-spool
+# Cleanup the collectd RRD (graphs) +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete
# Update DNS trust anchor %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem -- 2.30.2
Hey Michael,
Over the years I’ve used four different IPFire boxes. And during a "late-summer cleaning" I found lots of old RRD files. I ran this command: `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls`
And among those were lots of old unused openvpn graphs. I think I had 8-10 old openvpn graphs that were more than 1 year old.
I had started to write a wiki page for users about finding and deleting old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs. And then the wiki grew to ALL old graphs.
Below is a snipped version of my original `find`. I think there were ~40 total old obsolete RRD files.
``` [root@ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls 1838 304 -rw-r--r-- 1 root root 307712 Nov 17 2019 /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd 1794 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd 1821 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd . . . 1831 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd 1888 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd 1785 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd . . . 1791 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd 1880 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd 1875 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd 1874 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd 1781 152 -rw-r--r-- 1 root root 154888 Jan 13 2020 /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd 1937 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.105.rrd 1938 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.147.rrd 1935 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.194.rrd [root@ipfire ~] # ```
As you can see some of the files were left over from 2017 and they did not get cleaned out during hardware changes. I obviously did not follow the migrations wiki!
The wiki page was to be instructions for users to `find` the files and then to delete the files via the command line. While is was writing the wiki I remembered a Dev Mailing list message about sending users to the shell partially defeats the purpose of IPFire (bad paraphrase!). So instead of writing a wiki I wrote it a weekly fcronjob to the help everyone.
More than you ever wanted to know! Hope this helps!
Jon
On Aug 25, 2022, at 5:48 AM, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Jon,
Thanks for the patch.
Sorry if this sounds like a stupid question, but what is the rationale for this?
-Michael
On 25 Aug 2022, at 01:31, Jon Murphy jon.murphy@ipfire.org wrote:
- Created (mostly) for old openvpn graphs
- RRD removed when no graph modification for +365 days
- chosen since graph max out is 365 days
- fcron job runs once per week
- chosen since this is just a cleanup and it doesnt need to run everyday
Note: logging can be added if needed.
Signed-off-by: Jon Murphy jon.murphy@ipfire.org
config/cron/crontab | 3 +++ 1 file changed, 3 insertions(+)
diff --git a/config/cron/crontab b/config/cron/crontab index b727531fc..7088e0b75 100644 --- a/config/cron/crontab +++ b/config/cron/crontab @@ -80,5 +80,8 @@ HOME=/ # Cleanup the mail spool directory %weekly * * /usr/sbin/dma-cleanup-spool
+# Cleanup the collectd RRD (graphs) +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete
# Update DNS trust anchor %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem -- 2.30.2
Hello Jon,
On 25 Aug 2022, at 15:40, Jon Murphy jcmurphy26@gmail.com wrote:
Hey Michael,
Over the years I’ve used four different IPFire boxes. And during a "late-summer cleaning" I found lots of old RRD files. I ran this command: `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls`
Was this just because you had a look around in the file system, or did you suffer from low disk space?
And among those were lots of old unused openvpn graphs. I think I had 8-10 old openvpn graphs that were more than 1 year old.
I had started to write a wiki page for users about finding and deleting old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs. And then the wiki grew to ALL old graphs.
Below is a snipped version of my original `find`. I think there were ~40 total old obsolete RRD files.
[root@ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls 1838 304 -rw-r--r-- 1 root root 307712 Nov 17 2019 /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd 1794 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd 1821 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd . . . 1831 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd 1888 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd 1785 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd . . . 1791 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd 1880 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd 1875 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd 1874 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd 1781 152 -rw-r--r-- 1 root root 154888 Jan 13 2020 /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd 1937 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.105.rrd 1938 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.147.rrd 1935 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.194.rrd [root@ipfire ~] #As you can see some of the files were left over from 2017 and they did not get cleaned out during hardware changes. I obviously did not follow the migrations wiki!
Well, I suppose if it doesn’t happen automatically, we cannot rely on this :) I never follow this either.
The wiki page was to be instructions for users to `find` the files and then to delete the files via the command line. While is was writing the wiki I remembered a Dev Mailing list message about sending users to the shell partially defeats the purpose of IPFire (bad paraphrase!). So instead of writing a wiki I wrote it a weekly fcronjob to the help everyone.
More than you ever wanted to know! Hope this helps!
Yes it does, but in the end my concern remains:
Is it not possible that an OpenVPN connection is only being used once a year or something like that? Would we then not wipe any data? Wouldn’t those databases come back again?
I suppose a database that has not been touched in over a year would not be carrying any information because that would have been rotated out of it (because that is how RRD databases work). So maybe that can go. But I would argue only then. So we would have to check what the longest time is those databases retain any information.
Personally, I would redesign this whole thing a little. I would make them store a lot more data and I would even look for more metrics to collect. So, the extra space for a RRD database wouldn’t really bother me, even if the whole directory is going to use up gigabytes. If that is a problem for other users, I would like to know.
Best, -Michael
Jon
On Aug 25, 2022, at 5:48 AM, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Jon,
Thanks for the patch.
Sorry if this sounds like a stupid question, but what is the rationale for this?
-Michael
On 25 Aug 2022, at 01:31, Jon Murphy jon.murphy@ipfire.org wrote:
- Created (mostly) for old openvpn graphs
- RRD removed when no graph modification for +365 days
- chosen since graph max out is 365 days
- fcron job runs once per week
- chosen since this is just a cleanup and it doesnt need to run everyday
Note: logging can be added if needed.
Signed-off-by: Jon Murphy jon.murphy@ipfire.org
config/cron/crontab | 3 +++ 1 file changed, 3 insertions(+)
diff --git a/config/cron/crontab b/config/cron/crontab index b727531fc..7088e0b75 100644 --- a/config/cron/crontab +++ b/config/cron/crontab @@ -80,5 +80,8 @@ HOME=/ # Cleanup the mail spool directory %weekly * * /usr/sbin/dma-cleanup-spool
+# Cleanup the collectd RRD (graphs) +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete
# Update DNS trust anchor %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem -- 2.30.2
Michael,
On Aug 25, 2022, at 10:44 AM, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Jon,
On 25 Aug 2022, at 15:40, Jon Murphy <jcmurphy26@gmail.com mailto:jcmurphy26@gmail.com> wrote:
Hey Michael,
Over the years I’ve used four different IPFire boxes. And during a "late-summer cleaning" I found lots of old RRD files. I ran this command: `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls`
Was this just because you had a look around in the file system, or did you suffer from low disk space?
Just looking…
And among those were lots of old unused openvpn graphs. I think I had 8-10 old openvpn graphs that were more than 1 year old.
I had started to write a wiki page for users about finding and deleting old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs. And then the wiki grew to ALL old graphs.
Below is a snipped version of my original `find`. I think there were ~40 total old obsolete RRD files.
[root@ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls 1838 304 -rw-r--r-- 1 root root 307712 Nov 17 2019 /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd 1794 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd 1821 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd . . . 1831 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd 1888 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd 1785 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd . . . 1791 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd 1880 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd 1875 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd 1874 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd 1781 152 -rw-r--r-- 1 root root 154888 Jan 13 2020 /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd 1937 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.105.rrd 1938 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.147.rrd 1935 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.194.rrd [root@ipfire ~] #As you can see some of the files were left over from 2017 and they did not get cleaned out during hardware changes. I obviously did not follow the migrations wiki!
Well, I suppose if it doesn’t happen automatically, we cannot rely on this :) I never follow this either.
The wiki page was to be instructions for users to `find` the files and then to delete the files via the command line. While is was writing the wiki I remembered a Dev Mailing list message about sending users to the shell partially defeats the purpose of IPFire (bad paraphrase!). So instead of writing a wiki I wrote it a weekly fcronjob to the help everyone.
More than you ever wanted to know! Hope this helps!
Yes it does, but in the end my concern remains:
Is it not possible that an OpenVPN connection is only being used once a year or something like that? Would we then not wipe any data? Wouldn’t those databases come back again?
I will test by wiping the current data and then connecting via OpenVPN to see if the databases and graphs recover.
I suppose a database that has not been touched in over a year would not be carrying any information because that would have been rotated out of it (because that is how RRD databases work). So maybe that can go. But I would argue only then. So we would have to check what the longest time is those databases retain any information.
Who is the RRD guru for IPFire?
I think there is an RRD dump command. I will research.
Personally, I would redesign this whole thing a little. I would make them store a lot more data and I would even look for more metrics to collect. So, the extra space for a RRD database wouldn’t really bother me, even if the whole directory is going to use up gigabytes. If that is a problem for other users, I would like to know.
OK on my side! I am at 48% (Use%) for a 16 GB drive.
I’d like to see a periodic bandwidth test (speedtest). Once in awhile I drop to near 0 Mbps bandwidth. Then it is time to restart the cable gateway and maybe the local network.
Jon
Best, -Michael
Jon
On Aug 25, 2022, at 5:48 AM, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Jon,
Thanks for the patch.
Sorry if this sounds like a stupid question, but what is the rationale for this?
-Michael
On 25 Aug 2022, at 01:31, Jon Murphy jon.murphy@ipfire.org wrote:
- Created (mostly) for old openvpn graphs
- RRD removed when no graph modification for +365 days
- chosen since graph max out is 365 days
- fcron job runs once per week
- chosen since this is just a cleanup and it doesnt need to run everyday
Note: logging can be added if needed.
Signed-off-by: Jon Murphy jon.murphy@ipfire.org
config/cron/crontab | 3 +++ 1 file changed, 3 insertions(+)
diff --git a/config/cron/crontab b/config/cron/crontab index b727531fc..7088e0b75 100644 --- a/config/cron/crontab +++ b/config/cron/crontab @@ -80,5 +80,8 @@ HOME=/ # Cleanup the mail spool directory %weekly * * /usr/sbin/dma-cleanup-spool
+# Cleanup the collectd RRD (graphs) +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete
# Update DNS trust anchor %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem -- 2.30.2
Hello,
On 25 Aug 2022, at 17:17, Jon Murphy jcmurphy26@gmail.com wrote:
Michael,
On Aug 25, 2022, at 10:44 AM, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Jon,
On 25 Aug 2022, at 15:40, Jon Murphy jcmurphy26@gmail.com wrote:
Hey Michael,
Over the years I’ve used four different IPFire boxes. And during a "late-summer cleaning" I found lots of old RRD files. I ran this command: `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls`
Was this just because you had a look around in the file system, or did you suffer from low disk space?
Just looking…
:)
And among those were lots of old unused openvpn graphs. I think I had 8-10 old openvpn graphs that were more than 1 year old.
I had started to write a wiki page for users about finding and deleting old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs. And then the wiki grew to ALL old graphs.
Below is a snipped version of my original `find`. I think there were ~40 total old obsolete RRD files.
[root@ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls 1838 304 -rw-r--r-- 1 root root 307712 Nov 17 2019 /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd 1794 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd 1821 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd . . . 1831 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd 1888 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd 1785 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd . . . 1791 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd 1880 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd 1875 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd 1874 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd 1781 152 -rw-r--r-- 1 root root 154888 Jan 13 2020 /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd 1937 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.105.rrd 1938 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.147.rrd 1935 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.194.rrd [root@ipfire ~] #As you can see some of the files were left over from 2017 and they did not get cleaned out during hardware changes. I obviously did not follow the migrations wiki!
Well, I suppose if it doesn’t happen automatically, we cannot rely on this :) I never follow this either.
The wiki page was to be instructions for users to `find` the files and then to delete the files via the command line. While is was writing the wiki I remembered a Dev Mailing list message about sending users to the shell partially defeats the purpose of IPFire (bad paraphrase!). So instead of writing a wiki I wrote it a weekly fcronjob to the help everyone.
More than you ever wanted to know! Hope this helps!
Yes it does, but in the end my concern remains:
Is it not possible that an OpenVPN connection is only being used once a year or something like that? Would we then not wipe any data? Wouldn’t those databases come back again?
I will test by wiping the current data and then connecting via OpenVPN to see if the databases and graphs recover.
I am quite confident that should work just fine.
I suppose a database that has not been touched in over a year would not be carrying any information because that would have been rotated out of it (because that is how RRD databases work). So maybe that can go. But I would argue only then. So we would have to check what the longest time is those databases retain any information.
Who is the RRD guru for IPFire?
I think there is an RRD dump command. I will research.
rrdtool dump /var/log/rrd/collectd/localhost/entropy/entropy.rrd
That will show you a looooong list with all the data points.
Personally, I would redesign this whole thing a little. I would make them store a lot more data and I would even look for more metrics to collect. So, the extra space for a RRD database wouldn’t really bother me, even if the whole directory is going to use up gigabytes. If that is a problem for other users, I would like to know.
OK on my side! I am at 48% (Use%) for a 16 GB drive.
Yes, 16GB is perfectly fine.
I’d like to see a periodic bandwidth test (speedtest). Once in awhile I drop to near 0 Mbps bandwidth. Then it is time to restart the cable gateway and maybe the local network.
Yeah, that would be really nice to have :) I am just not sure how much traffic all IPFire devices like that would create.
Best, -Michael
Jon
Best, -Michael
Jon
On Aug 25, 2022, at 5:48 AM, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Jon,
Thanks for the patch.
Sorry if this sounds like a stupid question, but what is the rationale for this?
-Michael
On 25 Aug 2022, at 01:31, Jon Murphy jon.murphy@ipfire.org wrote:
- Created (mostly) for old openvpn graphs
- RRD removed when no graph modification for +365 days
- chosen since graph max out is 365 days
- fcron job runs once per week
- chosen since this is just a cleanup and it doesnt need to run everyday
Note: logging can be added if needed.
Signed-off-by: Jon Murphy jon.murphy@ipfire.org
config/cron/crontab | 3 +++ 1 file changed, 3 insertions(+)
diff --git a/config/cron/crontab b/config/cron/crontab index b727531fc..7088e0b75 100644 --- a/config/cron/crontab +++ b/config/cron/crontab @@ -80,5 +80,8 @@ HOME=/ # Cleanup the mail spool directory %weekly * * /usr/sbin/dma-cleanup-spool
+# Cleanup the collectd RRD (graphs) +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete
# Update DNS trust anchor %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem -- 2.30.2
Michael,
On Aug 25, 2022, at 11:32 AM, Michael Tremer michael.tremer@ipfire.org wrote:
Hello,
On 25 Aug 2022, at 17:17, Jon Murphy jcmurphy26@gmail.com wrote:
Michael,
On Aug 25, 2022, at 10:44 AM, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Jon,
On 25 Aug 2022, at 15:40, Jon Murphy jcmurphy26@gmail.com wrote:
Hey Michael,
Over the years I’ve used four different IPFire boxes. And during a "late-summer cleaning" I found lots of old RRD files. I ran this command: `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls`
Was this just because you had a look around in the file system, or did you suffer from low disk space?
Just looking…
:)
And among those were lots of old unused openvpn graphs. I think I had 8-10 old openvpn graphs that were more than 1 year old.
I had started to write a wiki page for users about finding and deleting old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs. And then the wiki grew to ALL old graphs.
Below is a snipped version of my original `find`. I think there were ~40 total old obsolete RRD files.
[root@ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls 1838 304 -rw-r--r-- 1 root root 307712 Nov 17 2019 /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd 1794 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd 1821 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd . . . 1831 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd 1888 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd 1785 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd . . . 1791 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd 1880 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd 1875 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd 1874 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd 1781 152 -rw-r--r-- 1 root root 154888 Jan 13 2020 /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd 1937 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.105.rrd 1938 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.147.rrd 1935 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.194.rrd [root@ipfire ~] #As you can see some of the files were left over from 2017 and they did not get cleaned out during hardware changes. I obviously did not follow the migrations wiki!
Well, I suppose if it doesn’t happen automatically, we cannot rely on this :) I never follow this either.
The wiki page was to be instructions for users to `find` the files and then to delete the files via the command line. While is was writing the wiki I remembered a Dev Mailing list message about sending users to the shell partially defeats the purpose of IPFire (bad paraphrase!). So instead of writing a wiki I wrote it a weekly fcronjob to the help everyone.
More than you ever wanted to know! Hope this helps!
Yes it does, but in the end my concern remains:
Is it not possible that an OpenVPN connection is only being used once a year or something like that? Would we then not wipe any data? Wouldn’t those databases come back again?
I will test by wiping the current data and then connecting via OpenVPN to see if the databases and graphs recover.
I am quite confident that should work just fine.
I suppose a database that has not been touched in over a year would not be carrying any information because that would have been rotated out of it (because that is how RRD databases work). So maybe that can go. But I would argue only then. So we would have to check what the longest time is those databases retain any information.
Who is the RRD guru for IPFire?
I think there is an RRD dump command. I will research.
rrdtool dump /var/log/rrd/collectd/localhost/entropy/entropy.rrd
That will show you a looooong list with all the data points.
Looks like it lasts a year!
Beginning of a section of round robin:
``` </database> </rra> <rra> <cf>AVERAGE</cf> <pdp_per_row>878</pdp_per_row> <!-- 26340 seconds -->
<params> <xff>1.0000000000e-01</xff> </params> <cdp_prep> <ds> <primary_value>2.5600000000e+02</primary_value> <secondary_value>2.5600000000e+02</secondary_value> <value>2.0326400000e+05</value> <unknown_datapoints>0</unknown_datapoints> </ds> </cdp_prep> <database> <!-- 2021-08-24 09:04:00 CDT / 1629813840 --> <row><v>3.3458068337e+03</v></row> <!-- 2021-08-24 16:23:00 CDT / 1629840180 --> <row><v>3.3946542141e+03</v></row> <!-- 2021-08-24 23:42:00 CDT / 1629866520 --> <row><v>3.3813018223e+03</v></row> <!-- 2021-08-25 07:01:00 CDT / 1629892860 --> <row><v>3.3501125285e+03</v></row>
```
End of the round robin:
``` <!-- 2022-08-23 09:10:00 CDT / 1661263800 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-23 16:29:00 CDT / 1661290140 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-23 23:48:00 CDT / 1661316480 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-24 07:07:00 CDT / 1661342820 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-24 14:26:00 CDT / 1661369160 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-24 21:45:00 CDT / 1661395500 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-25 05:04:00 CDT / 1661421840 --> <row><v>2.5600000000e+02</v></row> </database> </rra> <rra> <cf>MIN</cf> <pdp_per_row>878</pdp_per_row> <!-- 26340 seconds -->
```
Personally, I would redesign this whole thing a little. I would make them store a lot more data and I would even look for more metrics to collect. So, the extra space for a RRD database wouldn’t really bother me, even if the whole directory is going to use up gigabytes. If that is a problem for other users, I would like to know.
OK on my side! I am at 48% (Use%) for a 16 GB drive.
Yes, 16GB is perfectly fine.
I’d like to see a periodic bandwidth test (speedtest). Once in awhile I drop to near 0 Mbps bandwidth. Then it is time to restart the cable gateway and maybe the local network.
Yeah, that would be really nice to have :) I am just not sure how much traffic all IPFire devices like that would create.
Best, -Michael
Jon
Best, -Michael
Jon
On Aug 25, 2022, at 5:48 AM, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Jon,
Thanks for the patch.
Sorry if this sounds like a stupid question, but what is the rationale for this?
-Michael
On 25 Aug 2022, at 01:31, Jon Murphy jon.murphy@ipfire.org wrote:
- Created (mostly) for old openvpn graphs
- RRD removed when no graph modification for +365 days
- chosen since graph max out is 365 days
- fcron job runs once per week
- chosen since this is just a cleanup and it doesnt need to run everyday
Note: logging can be added if needed.
Signed-off-by: Jon Murphy jon.murphy@ipfire.org
config/cron/crontab | 3 +++ 1 file changed, 3 insertions(+)
diff --git a/config/cron/crontab b/config/cron/crontab index b727531fc..7088e0b75 100644 --- a/config/cron/crontab +++ b/config/cron/crontab @@ -80,5 +80,8 @@ HOME=/ # Cleanup the mail spool directory %weekly * * /usr/sbin/dma-cleanup-spool
+# Cleanup the collectd RRD (graphs) +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete
# Update DNS trust anchor %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem -- 2.30.2
Michael,
Last item below…
If OK please add your "Acked-by:" or "Reviewed-by:".
Jon
On Aug 25, 2022, at 11:47 AM, Jon Murphy jcmurphy26@gmail.com wrote:
Michael,
On Aug 25, 2022, at 11:32 AM, Michael Tremer <michael.tremer@ipfire.org mailto:michael.tremer@ipfire.org> wrote:
Hello,
On 25 Aug 2022, at 17:17, Jon Murphy <jcmurphy26@gmail.com mailto:jcmurphy26@gmail.com> wrote:
Michael,
On Aug 25, 2022, at 10:44 AM, Michael Tremer <michael.tremer@ipfire.org mailto:michael.tremer@ipfire.org> wrote:
Hello Jon,
On 25 Aug 2022, at 15:40, Jon Murphy <jcmurphy26@gmail.com mailto:jcmurphy26@gmail.com> wrote:
Hey Michael,
Over the years I’ve used four different IPFire boxes. And during a "late-summer cleaning" I found lots of old RRD files. I ran this command: `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls`
Was this just because you had a look around in the file system, or did you suffer from low disk space?
Just looking…
:)
And among those were lots of old unused openvpn graphs. I think I had 8-10 old openvpn graphs that were more than 1 year old.
I had started to write a wiki page for users about finding and deleting old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs. And then the wiki grew to ALL old graphs.
Below is a snipped version of my original `find`. I think there were ~40 total old obsolete RRD files.
[root@ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls 1838 304 -rw-r--r-- 1 root root 307712 Nov 17 2019 /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd 1794 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd 1821 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd . . . 1831 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd 1888 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd 1785 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd . . . 1791 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd 1880 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd 1875 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd 1874 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd 1781 152 -rw-r--r-- 1 root root 154888 Jan 13 2020 /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd 1937 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.105.rrd 1938 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.147.rrd 1935 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.194.rrd [root@ipfire ~] #As you can see some of the files were left over from 2017 and they did not get cleaned out during hardware changes. I obviously did not follow the migrations wiki!
Well, I suppose if it doesn’t happen automatically, we cannot rely on this :) I never follow this either.
The wiki page was to be instructions for users to `find` the files and then to delete the files via the command line. While is was writing the wiki I remembered a Dev Mailing list message about sending users to the shell partially defeats the purpose of IPFire (bad paraphrase!). So instead of writing a wiki I wrote it a weekly fcronjob to the help everyone.
More than you ever wanted to know! Hope this helps!
Yes it does, but in the end my concern remains:
Is it not possible that an OpenVPN connection is only being used once a year or something like that? Would we then not wipe any data? Wouldn’t those databases come back again?
I will test by wiping the current data and then connecting via OpenVPN to see if the databases and graphs recover.
I am quite confident that should work just fine.
I did a quick test and the folder and RRD file are both recreated.
Cool!
I suppose a database that has not been touched in over a year would not be carrying any information because that would have been rotated out of it (because that is how RRD databases work). So maybe that can go. But I would argue only then. So we would have to check what the longest time is those databases retain any information.
Who is the RRD guru for IPFire?
I think there is an RRD dump command. I will research.
rrdtool dump /var/log/rrd/collectd/localhost/entropy/entropy.rrd
That will show you a looooong list with all the data points.
Looks like it lasts a year!
Beginning of a section of round robin:
</database> </rra> <rra> <cf>AVERAGE</cf> <pdp_per_row>878</pdp_per_row> <!-- 26340 seconds --> <params> <xff>1.0000000000e-01</xff> </params> <cdp_prep> <ds> <primary_value>2.5600000000e+02</primary_value> <secondary_value>2.5600000000e+02</secondary_value> <value>2.0326400000e+05</value> <unknown_datapoints>0</unknown_datapoints> </ds> </cdp_prep> <database> <!-- 2021-08-24 09:04:00 CDT / 1629813840 --> <row><v>3.3458068337e+03</v></row> <!-- 2021-08-24 16:23:00 CDT / 1629840180 --> <row><v>3.3946542141e+03</v></row> <!-- 2021-08-24 23:42:00 CDT / 1629866520 --> <row><v>3.3813018223e+03</v></row> <!-- 2021-08-25 07:01:00 CDT / 1629892860 --> <row><v>3.3501125285e+03</v></row>End of the round robin:
<!-- 2022-08-23 09:10:00 CDT / 1661263800 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-23 16:29:00 CDT / 1661290140 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-23 23:48:00 CDT / 1661316480 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-24 07:07:00 CDT / 1661342820 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-24 14:26:00 CDT / 1661369160 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-24 21:45:00 CDT / 1661395500 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-25 05:04:00 CDT / 1661421840 --> <row><v>2.5600000000e+02</v></row> </database> </rra> <rra> <cf>MIN</cf> <pdp_per_row>878</pdp_per_row> <!-- 26340 seconds -->Personally, I would redesign this whole thing a little. I would make them store a lot more data and I would even look for more metrics to collect. So, the extra space for a RRD database wouldn’t really bother me, even if the whole directory is going to use up gigabytes. If that is a problem for other users, I would like to know.
OK on my side! I am at 48% (Use%) for a 16 GB drive.
Yes, 16GB is perfectly fine.
I’d like to see a periodic bandwidth test (speedtest). Once in awhile I drop to near 0 Mbps bandwidth. Then it is time to restart the cable gateway and maybe the local network.
Yeah, that would be really nice to have :) I am just not sure how much traffic all IPFire devices like that would create.
Best, -Michael
Jon
Best, -Michael
Jon
On Aug 25, 2022, at 5:48 AM, Michael Tremer <michael.tremer@ipfire.org mailto:michael.tremer@ipfire.org> wrote:
Hello Jon,
Thanks for the patch.
Sorry if this sounds like a stupid question, but what is the rationale for this?
-Michael
> On 25 Aug 2022, at 01:31, Jon Murphy <jon.murphy@ipfire.org mailto:jon.murphy@ipfire.org> wrote: > > - Created (mostly) for old openvpn graphs > - RRD removed when no graph modification for +365 days > - chosen since graph max out is 365 days > - fcron job runs once per week > - chosen since this is just a cleanup and it doesnt need to run everyday > Note: logging can be added if needed. > > Signed-off-by: Jon Murphy <jon.murphy@ipfire.org mailto:jon.murphy@ipfire.org> > --- > config/cron/crontab | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/config/cron/crontab b/config/cron/crontab > index b727531fc..7088e0b75 100644 > --- a/config/cron/crontab > +++ b/config/cron/crontab > @@ -80,5 +80,8 @@ HOME=/ > # Cleanup the mail spool directory > %weekly * * /usr/sbin/dma-cleanup-spool > > +# Cleanup the collectd RRD (graphs) > +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete > + > # Update DNS trust anchor > %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem > -- > 2.30.2
Bump / nudge... Is this OK?
Jon
On Aug 25, 2022, at 4:34 PM, Jon Murphy jcmurphy26@gmail.com wrote:
Michael,
Last item below…
If OK please add your "Acked-by:" or "Reviewed-by:".
Jon
On Aug 25, 2022, at 11:47 AM, Jon Murphy <jcmurphy26@gmail.com mailto:jcmurphy26@gmail.com> wrote:
Michael,
On Aug 25, 2022, at 11:32 AM, Michael Tremer <michael.tremer@ipfire.org mailto:michael.tremer@ipfire.org> wrote:
Hello,
On 25 Aug 2022, at 17:17, Jon Murphy <jcmurphy26@gmail.com mailto:jcmurphy26@gmail.com> wrote:
Michael,
On Aug 25, 2022, at 10:44 AM, Michael Tremer <michael.tremer@ipfire.org mailto:michael.tremer@ipfire.org> wrote:
Hello Jon,
On 25 Aug 2022, at 15:40, Jon Murphy <jcmurphy26@gmail.com mailto:jcmurphy26@gmail.com> wrote:
Hey Michael,
Over the years I’ve used four different IPFire boxes. And during a "late-summer cleaning" I found lots of old RRD files. I ran this command: `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls`
Was this just because you had a look around in the file system, or did you suffer from low disk space?
Just looking…
:)
And among those were lots of old unused openvpn graphs. I think I had 8-10 old openvpn graphs that were more than 1 year old.
I had started to write a wiki page for users about finding and deleting old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs. And then the wiki grew to ALL old graphs.
Below is a snipped version of my original `find`. I think there were ~40 total old obsolete RRD files.
[root@ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls 1838 304 -rw-r--r-- 1 root root 307712 Nov 17 2019 /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd 1794 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd 1821 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd . . . 1831 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd 1888 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd 1785 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd . . . 1791 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd 1880 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd 1875 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd 1874 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd 1781 152 -rw-r--r-- 1 root root 154888 Jan 13 2020 /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd 1937 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.105.rrd 1938 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.147.rrd 1935 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.194.rrd [root@ipfire ~] #As you can see some of the files were left over from 2017 and they did not get cleaned out during hardware changes. I obviously did not follow the migrations wiki!
Well, I suppose if it doesn’t happen automatically, we cannot rely on this :) I never follow this either.
The wiki page was to be instructions for users to `find` the files and then to delete the files via the command line. While is was writing the wiki I remembered a Dev Mailing list message about sending users to the shell partially defeats the purpose of IPFire (bad paraphrase!). So instead of writing a wiki I wrote it a weekly fcronjob to the help everyone.
More than you ever wanted to know! Hope this helps!
Yes it does, but in the end my concern remains:
Is it not possible that an OpenVPN connection is only being used once a year or something like that? Would we then not wipe any data? Wouldn’t those databases come back again?
I will test by wiping the current data and then connecting via OpenVPN to see if the databases and graphs recover.
I am quite confident that should work just fine.
I did a quick test and the folder and RRD file are both recreated.
Cool!
I suppose a database that has not been touched in over a year would not be carrying any information because that would have been rotated out of it (because that is how RRD databases work). So maybe that can go. But I would argue only then. So we would have to check what the longest time is those databases retain any information.
Who is the RRD guru for IPFire?
I think there is an RRD dump command. I will research.
rrdtool dump /var/log/rrd/collectd/localhost/entropy/entropy.rrd
That will show you a looooong list with all the data points.
Looks like it lasts a year!
Beginning of a section of round robin:
</database> </rra> <rra> <cf>AVERAGE</cf> <pdp_per_row>878</pdp_per_row> <!-- 26340 seconds --> <params> <xff>1.0000000000e-01</xff> </params> <cdp_prep> <ds> <primary_value>2.5600000000e+02</primary_value> <secondary_value>2.5600000000e+02</secondary_value> <value>2.0326400000e+05</value> <unknown_datapoints>0</unknown_datapoints> </ds> </cdp_prep> <database> <!-- 2021-08-24 09:04:00 CDT / 1629813840 --> <row><v>3.3458068337e+03</v></row> <!-- 2021-08-24 16:23:00 CDT / 1629840180 --> <row><v>3.3946542141e+03</v></row> <!-- 2021-08-24 23:42:00 CDT / 1629866520 --> <row><v>3.3813018223e+03</v></row> <!-- 2021-08-25 07:01:00 CDT / 1629892860 --> <row><v>3.3501125285e+03</v></row>End of the round robin:
<!-- 2022-08-23 09:10:00 CDT / 1661263800 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-23 16:29:00 CDT / 1661290140 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-23 23:48:00 CDT / 1661316480 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-24 07:07:00 CDT / 1661342820 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-24 14:26:00 CDT / 1661369160 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-24 21:45:00 CDT / 1661395500 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-25 05:04:00 CDT / 1661421840 --> <row><v>2.5600000000e+02</v></row> </database> </rra> <rra> <cf>MIN</cf> <pdp_per_row>878</pdp_per_row> <!-- 26340 seconds -->Personally, I would redesign this whole thing a little. I would make them store a lot more data and I would even look for more metrics to collect. So, the extra space for a RRD database wouldn’t really bother me, even if the whole directory is going to use up gigabytes. If that is a problem for other users, I would like to know.
OK on my side! I am at 48% (Use%) for a 16 GB drive.
Yes, 16GB is perfectly fine.
I’d like to see a periodic bandwidth test (speedtest). Once in awhile I drop to near 0 Mbps bandwidth. Then it is time to restart the cable gateway and maybe the local network.
Yeah, that would be really nice to have :) I am just not sure how much traffic all IPFire devices like that would create.
Best, -Michael
Jon
Best, -Michael
Jon
> On Aug 25, 2022, at 5:48 AM, Michael Tremer <michael.tremer@ipfire.org mailto:michael.tremer@ipfire.org> wrote: > > Hello Jon, > > Thanks for the patch. > > Sorry if this sounds like a stupid question, but what is the rationale for this? > > -Michael > >> On 25 Aug 2022, at 01:31, Jon Murphy <jon.murphy@ipfire.org mailto:jon.murphy@ipfire.org> wrote: >> >> - Created (mostly) for old openvpn graphs >> - RRD removed when no graph modification for +365 days >> - chosen since graph max out is 365 days >> - fcron job runs once per week >> - chosen since this is just a cleanup and it doesnt need to run everyday >> Note: logging can be added if needed. >> >> Signed-off-by: Jon Murphy <jon.murphy@ipfire.org mailto:jon.murphy@ipfire.org> >> --- >> config/cron/crontab | 3 +++ >> 1 file changed, 3 insertions(+) >> >> diff --git a/config/cron/crontab b/config/cron/crontab >> index b727531fc..7088e0b75 100644 >> --- a/config/cron/crontab >> +++ b/config/cron/crontab >> @@ -80,5 +80,8 @@ HOME=/ >> # Cleanup the mail spool directory >> %weekly * * /usr/sbin/dma-cleanup-spool >> >> +# Cleanup the collectd RRD (graphs) >> +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete >> + >> # Update DNS trust anchor >> %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem >> -- >> 2.30.2
I think this is fine for me:
Reviewed-by: Michael Tremer michael.tremer@ipfire.org
Thanks for the nudge.
-Michael
On 20 Sep 2022, at 03:28, Jon Murphy jcmurphy26@gmail.com wrote:
Bump / nudge... Is this OK?
Jon
On Aug 25, 2022, at 4:34 PM, Jon Murphy jcmurphy26@gmail.com wrote:
Michael,
Last item below…
If OK please add your "Acked-by:" or "Reviewed-by:".
Jon
On Aug 25, 2022, at 11:47 AM, Jon Murphy jcmurphy26@gmail.com wrote:
Michael,
On Aug 25, 2022, at 11:32 AM, Michael Tremer michael.tremer@ipfire.org wrote:
Hello,
On 25 Aug 2022, at 17:17, Jon Murphy jcmurphy26@gmail.com wrote:
Michael,
On Aug 25, 2022, at 10:44 AM, Michael Tremer michael.tremer@ipfire.org wrote:
Hello Jon,
> On 25 Aug 2022, at 15:40, Jon Murphy jcmurphy26@gmail.com wrote: > > Hey Michael, > > Over the years I’ve used four different IPFire boxes. And during a "late-summer cleaning" I found lots of old RRD files. I ran this command: > `find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls`
Was this just because you had a look around in the file system, or did you suffer from low disk space?
Just looking…
:)
> And among those were lots of old unused openvpn graphs. I think I had 8-10 old openvpn graphs that were more than 1 year old. > > I had started to write a wiki page for users about finding and deleting old `/var/log/rrd/collectd/localhost/openvpn-*` RRDs. And then the wiki grew to ALL old graphs. > > Below is a snipped version of my original `find`. I think there were ~40 total old obsolete RRD files. > > ``` > [root@ipfire ~] # find /var/log/rrd -mtime +365 -type f -name '*.rrd' -ls > 1838 304 -rw-r--r-- 1 root root 307712 Nov 17 2019 /var/log/rrd/collectd/localhost/interface/if_octets-dummy0.rrd > 1794 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts0-virtual-0/temperature-temp1.rrd > 1821 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/temperature-temp2.rrd > . . . > 1831 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-it8721-isa-0a30/fanspeed-fan3.rrd > 1888 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-soc_dts1-virtual-0/temperature-temp1.rrd > 1785 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/temperature-temp2.rrd > . . . > 1791 152 -rw-r--r-- 1 root root 154888 Aug 12 2017 /var/log/rrd/collectd/localhost/sensors-smsc47b397-isa-0480/fanspeed-fan3.rrd > 1880 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/thermal-thermal_zone1/temperature-temperature.rrd > 1875 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp4.rrd > 1874 152 -rw-r--r-- 1 root root 154888 Jul 18 2021 /var/log/rrd/collectd/localhost/sensors-coretemp-isa-0000/temperature-temp5.rrd > 1781 152 -rw-r--r-- 1 root root 154888 Jan 13 2020 /var/log/rrd/collectd/localhost/thermal-thermal_zone2/temperature-temperature.rrd > 1937 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.105.rrd > 1938 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.147.rrd > 1935 56 -rw-r--r-- 1 nobody nobody 56800 Jul 18 2021 /var/log/rrd/iptraffic/192.168.60.194.rrd > [root@ipfire ~] # > ``` > > As you can see some of the files were left over from 2017 and they did not get cleaned out during hardware changes. I obviously did not follow the migrations wiki!
Well, I suppose if it doesn’t happen automatically, we cannot rely on this :) I never follow this either.
> The wiki page was to be instructions for users to `find` the files and then to delete the files via the command line. While is was writing the wiki I remembered a Dev Mailing list message about sending users to the shell partially defeats the purpose of IPFire (bad paraphrase!). So instead of writing a wiki I wrote it a weekly fcronjob to the help everyone.
> More than you ever wanted to know! Hope this helps!
Yes it does, but in the end my concern remains:
Is it not possible that an OpenVPN connection is only being used once a year or something like that? Would we then not wipe any data? Wouldn’t those databases come back again?
I will test by wiping the current data and then connecting via OpenVPN to see if the databases and graphs recover.
I am quite confident that should work just fine.
I did a quick test and the folder and RRD file are both recreated.
Cool!
I suppose a database that has not been touched in over a year would not be carrying any information because that would have been rotated out of it (because that is how RRD databases work). So maybe that can go. But I would argue only then. So we would have to check what the longest time is those databases retain any information.
Who is the RRD guru for IPFire?
I think there is an RRD dump command. I will research.
rrdtool dump /var/log/rrd/collectd/localhost/entropy/entropy.rrd
That will show you a looooong list with all the data points.
Looks like it lasts a year!
Beginning of a section of round robin:
</database> </rra> <rra> <cf>AVERAGE</cf> <pdp_per_row>878</pdp_per_row> <!-- 26340 seconds --> <params> <xff>1.0000000000e-01</xff> </params> <cdp_prep> <ds> <primary_value>2.5600000000e+02</primary_value> <secondary_value>2.5600000000e+02</secondary_value> <value>2.0326400000e+05</value> <unknown_datapoints>0</unknown_datapoints> </ds> </cdp_prep> <database> <!-- 2021-08-24 09:04:00 CDT / 1629813840 --> <row><v>3.3458068337e+03</v></row> <!-- 2021-08-24 16:23:00 CDT / 1629840180 --> <row><v>3.3946542141e+03</v></row> <!-- 2021-08-24 23:42:00 CDT / 1629866520 --> <row><v>3.3813018223e+03</v></row> <!-- 2021-08-25 07:01:00 CDT / 1629892860 --> <row><v>3.3501125285e+03</v></row>End of the round robin:
<!-- 2022-08-23 09:10:00 CDT / 1661263800 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-23 16:29:00 CDT / 1661290140 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-23 23:48:00 CDT / 1661316480 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-24 07:07:00 CDT / 1661342820 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-24 14:26:00 CDT / 1661369160 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-24 21:45:00 CDT / 1661395500 --> <row><v>2.5600000000e+02</v></row> <!-- 2022-08-25 05:04:00 CDT / 1661421840 --> <row><v>2.5600000000e+02</v></row> </database> </rra> <rra> <cf>MIN</cf> <pdp_per_row>878</pdp_per_row> <!-- 26340 seconds -->Personally, I would redesign this whole thing a little. I would make them store a lot more data and I would even look for more metrics to collect. So, the extra space for a RRD database wouldn’t really bother me, even if the whole directory is going to use up gigabytes. If that is a problem for other users, I would like to know.
OK on my side! I am at 48% (Use%) for a 16 GB drive.
Yes, 16GB is perfectly fine.
I’d like to see a periodic bandwidth test (speedtest). Once in awhile I drop to near 0 Mbps bandwidth. Then it is time to restart the cable gateway and maybe the local network.
Yeah, that would be really nice to have :) I am just not sure how much traffic all IPFire devices like that would create.
Best, -Michael
Jon
Best, -Michael
> Jon > > >> On Aug 25, 2022, at 5:48 AM, Michael Tremer michael.tremer@ipfire.org wrote: >> >> Hello Jon, >> >> Thanks for the patch. >> >> Sorry if this sounds like a stupid question, but what is the rationale for this? >> >> -Michael >> >>> On 25 Aug 2022, at 01:31, Jon Murphy jon.murphy@ipfire.org wrote: >>> >>> - Created (mostly) for old openvpn graphs >>> - RRD removed when no graph modification for +365 days >>> - chosen since graph max out is 365 days >>> - fcron job runs once per week >>> - chosen since this is just a cleanup and it doesnt need to run everyday >>> Note: logging can be added if needed. >>> >>> Signed-off-by: Jon Murphy jon.murphy@ipfire.org >>> --- >>> config/cron/crontab | 3 +++ >>> 1 file changed, 3 insertions(+) >>> >>> diff --git a/config/cron/crontab b/config/cron/crontab >>> index b727531fc..7088e0b75 100644 >>> --- a/config/cron/crontab >>> +++ b/config/cron/crontab >>> @@ -80,5 +80,8 @@ HOME=/ >>> # Cleanup the mail spool directory >>> %weekly * * /usr/sbin/dma-cleanup-spool >>> >>> +# Cleanup the collectd RRD (graphs) >>> +%weekly * * /bin/find /var/log/rrd -mtime +365 -type f -name '*.rrd' -delete -o -type d -empty -delete >>> + >>> # Update DNS trust anchor >>> %daily,random * * @runas(nobody) /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem >>> -- >>> 2.30.2
-
Bernhard Bitsch -
Jon Murphy -
Jon Murphy -
Michael Tremer