Re: [PATCH] rsnapshot: New addon

bump…

(don’t forget about me!)

On Apr 11, 2023, at 2:30 PM, Jon Murphy jon.murphy@ipfire.org wrote:

• What is it? rsnapshot is a filesystem snapshot utility based on rsync. rsnapshot makes it easy to make periodic snapshots of the ipfire device. The code makes extensive use of hard links whenever possible, to greatly reduce the disk space required. See: https://rsnapshot.org

• Why is it needed? Rsnapshot backups run multiple times per day (e.g., once per day up to 24 times per day). Rsnapshot is much easier to configure, setup and use than the borg backup add-on. (I found borg somewhat confusing). Rsnapshot completes each backup very fast. Unlike borg, rsnapshot does not compress each backup before storage. During a complete rebuild, borg backup need installation of the borg add-on to recover archived files. Rsnapshot backups can be copied directly from the backup drive. Current backups (backup.pl or borg) could corrupt sqlite3 databases by running a backup during a database write. This add-on includes a script specifically for sqlite backups.

• IPFire Wiki In process at: https://wiki.ipfire.org/addons/rsnapshot

Thanks to Gerd for creating a first build and a nice template for me!

Signed-off-by: Jon Murphy jon.murphy@ipfire.org

config/backup/includes/rsnapshot | 1 + config/rootfiles/packages/rsnapshot | 17 +++ config/rsnapshot/backup_sqlite.sh | 40 +++++ config/rsnapshot/rsnapshot-daily | 8 + config/rsnapshot/rsnapshot-hourly | 8 + config/rsnapshot/rsnapshot-monthly | 8 + config/rsnapshot/rsnapshot-weekly | 8 + config/rsnapshot/rsnapshot-yearly | 8 + config/rsnapshot/rsnapshot.conf | 218 ++++++++++++++++++++++++++++ lfs/rsnapshot | 114 +++++++++++++++ make.sh | 1 + 11 files changed, 431 insertions(+) create mode 100644 config/backup/includes/rsnapshot create mode 100644 config/rootfiles/packages/rsnapshot create mode 100644 config/rsnapshot/backup_sqlite.sh create mode 100644 config/rsnapshot/rsnapshot-daily create mode 100644 config/rsnapshot/rsnapshot-hourly create mode 100644 config/rsnapshot/rsnapshot-monthly create mode 100644 config/rsnapshot/rsnapshot-weekly create mode 100644 config/rsnapshot/rsnapshot-yearly create mode 100644 config/rsnapshot/rsnapshot.conf create mode 100644 lfs/rsnapshot

diff --git a/config/backup/includes/rsnapshot b/config/backup/includes/rsnapshot new file mode 100644 index 000000000..4d1b48a5a --- /dev/null +++ b/config/backup/includes/rsnapshot @@ -0,0 +1 @@ +/etc/rsnapshot.conf diff --git a/config/rootfiles/packages/rsnapshot b/config/rootfiles/packages/rsnapshot new file mode 100644 index 000000000..d907a615f --- /dev/null +++ b/config/rootfiles/packages/rsnapshot @@ -0,0 +1,17 @@ +etc/fcron.daily/rsnapshot-daily +etc/fcron.hourly/rsnapshot-hourly +etc/fcron.monthly/rsnapshot-monthly +etc/fcron.weekly/rsnapshot-weekly +etc/rsnapshot.conf +etc/rsnapshot.conf.default +usr/bin/rsnapshot +usr/bin/rsnapshot-diff +#usr/share/man/man1/rsnapshot-diff.1 +#usr/share/man/man1/rsnapshot.1 +var/ipfire/backup/addons/includes/rsnapshot +var/ipfire/backup/bin/backup_sqlite.sh +var/ipfire/backup/bin/rsnapshot-daily +var/ipfire/backup/bin/rsnapshot-hourly +var/ipfire/backup/bin/rsnapshot-monthly +var/ipfire/backup/bin/rsnapshot-weekly +var/ipfire/backup/bin/rsnapshot-yearly diff --git a/config/rsnapshot/backup_sqlite.sh b/config/rsnapshot/backup_sqlite.sh new file mode 100644 index 000000000..3ed54e49d --- /dev/null +++ b/config/rsnapshot/backup_sqlite.sh @@ -0,0 +1,40 @@ +#!/bin/bash +set -e +set -u +#set -x

+############################################################################## +# backup_sqlite.sh +# +# http://www.rsnapshot.org/ +# +# This is a simple shell script to backup a sqlite database with rsnapshot. +# +# This script simply needs to dump a file into the current working directory. +# rsnapshot handles everything else. +# +# The assumption is that this will be invoked from rsnapshot. +# See: +# https://rsnapshot.org/rsnapshot/docs/docbook/rest.html#backup-script +# +# Please remember that these backup scripts will be invoked as the user +# running rsnapshot. Make sure your backup scripts are owned by root, +# and not writable by anyone else. +# If you fail to do this, anyone with write access to these backup scripts +# will be able to put commands in them that will be run as the root user. +# If they are malicious, they could take over your server. +# +# chown root:root backup_sqlite.sh +# chmod 700 backup_sqlite.sh +# +##############################################################################

+umask 0077

+# backup the database +/bin/find /var -iname *.db -exec bash -c ' /usr/bin/file {} | /bin/grep -q "SQLite 3" && /usr/bin/sqlite3 {} ".backup $(/usr/bin/basename {})" ' ;

+# make the backup readable only by root +#/bin/chmod 600 filename.db

+exit diff --git a/config/rsnapshot/rsnapshot-daily b/config/rsnapshot/rsnapshot-daily new file mode 100644 index 000000000..99e560adc --- /dev/null +++ b/config/rsnapshot/rsnapshot-daily @@ -0,0 +1,8 @@ +#!/bin/bash +set -e

+CONFIG=/etc/rsnapshot.conf

+/usr/bin/rsnapshot -c $CONFIG daily

+exit diff --git a/config/rsnapshot/rsnapshot-hourly b/config/rsnapshot/rsnapshot-hourly new file mode 100644 index 000000000..8dc2cd75f --- /dev/null +++ b/config/rsnapshot/rsnapshot-hourly @@ -0,0 +1,8 @@ +#!/bin/bash +set -e

+CONFIG=/etc/rsnapshot.conf

+/usr/bin/rsnapshot -c $CONFIG sync && /usr/bin/rsnapshot -c $CONFIG hourly

+exit diff --git a/config/rsnapshot/rsnapshot-monthly b/config/rsnapshot/rsnapshot-monthly new file mode 100644 index 000000000..9c658250d --- /dev/null +++ b/config/rsnapshot/rsnapshot-monthly @@ -0,0 +1,8 @@ +#!/bin/bash +set -e

+CONFIG=/etc/rsnapshot.conf

+/usr/bin/rsnapshot -c $CONFIG monthly

+exit diff --git a/config/rsnapshot/rsnapshot-weekly b/config/rsnapshot/rsnapshot-weekly new file mode 100644 index 000000000..f5b18530d --- /dev/null +++ b/config/rsnapshot/rsnapshot-weekly @@ -0,0 +1,8 @@ +#!/bin/bash +set -e

+CONFIG=/etc/rsnapshot.conf

+/usr/bin/rsnapshot -c $CONFIG weekly

+exit diff --git a/config/rsnapshot/rsnapshot-yearly b/config/rsnapshot/rsnapshot-yearly new file mode 100644 index 000000000..218e9ee84 --- /dev/null +++ b/config/rsnapshot/rsnapshot-yearly @@ -0,0 +1,8 @@ +#!/bin/bash +set -e

+CONFIG=/etc/rsnapshot.conf

+/usr/bin/rsnapshot -c $CONFIG yearly

+exit diff --git a/config/rsnapshot/rsnapshot.conf b/config/rsnapshot/rsnapshot.conf new file mode 100644 index 000000000..cdec0404b --- /dev/null +++ b/config/rsnapshot/rsnapshot.conf @@ -0,0 +1,218 @@ +################################################# +# rsnapshot.conf - rsnapshot configuration file # +################################################# +# # +# PLEASE BE AWARE OF THE FOLLOWING RULE: # +# # +# This file requires tabs between elements # +# # +#################################################

+####################### +# CONFIG FILE VERSION # +#######################

+config_version 1.2

+########################### +# SNAPSHOT ROOT DIRECTORY # +###########################

+# All snapshots will be stored under this root directory. +# +snapshot_root /mnt/hdd/snapshots/

+# If no_create_root is enabled, rsnapshot will not automatically create the +# snapshot_root directory. This is particularly useful if you are backing +# up to removable media, such as a FireWire or USB drive. +# +no_create_root 1

+################################# +# EXTERNAL PROGRAM DEPENDENCIES # +#################################

+# LINUX USERS: Be sure to uncomment "cmd_cp". This gives you extra features. +# EVERYONE ELSE: Leave "cmd_cp" commented out for compatibility. +# +# See the README file or the man page for more details. +# +cmd_cp /bin/cp

+# uncomment this to use the rm program instead of the built-in perl routine. +# +cmd_rm /bin/rm

+# rsync must be enabled for anything to work. This is the only command that +# must be enabled. +# +cmd_rsync /usr/bin/rsync

+# Uncomment this to enable remote ssh backups over rsync. +# +#cmd_ssh /usr/bin/ssh

+# Comment this out to disable syslog support. +# +cmd_logger /usr/bin/logger

+# Uncomment this to specify the path to "du" for disk usage checks. +# If you have an older version of "du", you may also want to check the +# "du_args" parameter below. +# +cmd_du /usr/bin/du

+# Uncomment this to specify the path to rsnapshot-diff. +# +cmd_rsnapshot_diff /usr/bin/rsnapshot-diff

+# Specify the path to a script (and any optional arguments) to run right +# before rsnapshot syncs files +# +#cmd_preexec /path/to/preexec/script

+# Specify the path to a script (and any optional arguments) to run right +# after rsnapshot syncs files +# +#cmd_postexec /path/to/postexec/script

+######################################### +# BACKUP LEVELS / INTERVALS # +# Must be unique and in ascending order # +# e.g. alpha, beta, gamma, etc. # +#########################################

+retain hourly 2 # 2 backups in one day (every 12 hours) +#retain hourly 24 # 24 backups in one day +retain daily 7 # 7 backups in one week +retain weekly 4 # 4 backups in one month +retain monthly 12 # 12 backups in one year +retain yearly 2 # 2 backups total

+#retain alpha 6 # 6 backups in one beta +#retain beta 7 # 7 backups in one gamma +#retain gamma 4 # you get the idea! +#retain delta 3

+############################################ +# GLOBAL OPTIONS # +# All are optional, with sensible defaults # +############################################

+# Verbose level, 1 through 5. +# 1 Quiet Print fatal errors only +# 2 Default Print errors and warnings only +# 3 Verbose Show equivalent shell commands being executed +# 4 Extra Verbose Show extra verbose information +# 5 Debug mode Everything +# +verbose 2

+# Same as "verbose" above, but controls the amount of data sent to the +# logfile, if one is being used. The default is 3. +# +loglevel 2

+# If you enable this, data will be written to the file you specify. The +# amount of data written is controlled by the "loglevel" parameter. +# +#logfile /var/log/rsnapshot.log

+# If enabled, rsnapshot will write a lockfile to prevent two instances +# from running simultaneously (and messing up the snapshot_root). +# If you enable this, make sure the lockfile directory is not world +# writable. Otherwise anyone can prevent the program from running. +# +lockfile /run/rsnapshot.pid

+# By default, rsnapshot check lockfile, check if PID is running +# and if not, consider lockfile as stale, then start +# Enabling this stop rsnapshot if PID in lockfile is not running +# +#stop_on_stale_lockfile 0

+# Default rsync args. All rsync commands have at least these options set. +# +# Note: This is the exception to the rule of "requires tabs between elements" +# tab between "rsync_long_args" and "--delete" +# space between all other arguments +# +# rsync_long_args<tab>--delete<space>--numeric-ids<space>--relative +# +#rsync_short_args -a +#rsync_long_args --delete --numeric-ids --relative --delete-excluded

+# ssh has no args passed by default, but you can specify some here. +# +#ssh_args -p 22

+# Default arguments for the "du" program (for disk space reporting). +# The GNU version of "du" is preferred. See the man page for more details. +# If your version of "du" doesn't support the -h flag, try -k flag instead. +# +#du_args -csh

+# If this is enabled, rsync won't span filesystem partitions within a +# backup point. This essentially passes the -x option to rsync. +# The default is 0 (off). +# +#one_fs 0

+# The include and exclude parameters, if enabled, simply get passed directly +# to rsync. If you have multiple include/exclude patterns, put each one on a +# separate line. Please look up the --include and --exclude options in the +# rsync man page for more details on how to specify file name patterns. +# +exclude .cache/ +exclude /var/cache +#include ??? +#exclude ???

+# The include_file and exclude_file parameters, if enabled, simply get +# passed directly to rsync. Please look up the --include-from and +# --exclude-from options in the rsync man page for more details. +# +#include_file /path/to/include/file +#exclude_file /path/to/exclude/file

+# If your version of rsync supports --link-dest, consider enabling this. +# This is the best way to support special files (FIFOs, etc) cross-platform. +# The default is 0 (off). +# +link_dest 1

+# When sync_first is enabled, it changes the default behaviour of rsnapshot. +# Normally, when rsnapshot is called with its lowest interval +# (i.e.: "rsnapshot alpha"), it will sync files AND rotate the lowest +# intervals. With sync_first enabled, "rsnapshot sync" handles the file sync, +# and all interval calls simply rotate files. See the man page for more +# details. The default is 0 (off). +# +sync_first 1

+# If enabled, rsnapshot will move the oldest directory for each interval +# to [interval_name].delete, then it will remove the lockfile and delete +# that directory just before it exits. The default is 0 (off). +# +#use_lazy_deletes 0

+# Number of rsync re-tries. If you experience any network problems or +# network card issues that tend to cause ssh to fail with errors like +# "Corrupted MAC on input", for example, set this to a non-zero value +# to have the rsync operation re-tried. +# +#rsync_numtries 0

+############################### +### BACKUP POINTS / SCRIPTS ### +###############################

+# LOCALHOST +backup /etc/ localhost/ +backup /var/ localhost/ +backup /root/ localhost/

+backup_script /var/ipfire/backup/bin/backup_sqlite.sh localhost/sqlite3/

diff --git a/lfs/rsnapshot b/lfs/rsnapshot new file mode 100644 index 000000000..174eff452 --- /dev/null +++ b/lfs/rsnapshot @@ -0,0 +1,114 @@ +############################################################################### +# # +# IPFire.org - A linux based firewall # +# Copyright (C) 2023 IPFire Team info@ipfire.org # +# # +# This program is free software: you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation, either version 3 of the License, or # +# (at your option) any later version. # +# # +# This program is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with this program. If not, see http://www.gnu.org/licenses/. # +# # +###############################################################################

+############################################################################### +# Definitions +###############################################################################

+include Config

+SUMMARY = filesystem backup & snapshot utility

+VER = 1.4.5

+THISAPP = rsnapshot-$(VER) +DL_FILE = $(THISAPP).tar.gz +DL_FROM = $(URL_IPFIRE) +DIR_APP = $(DIR_SRC)/$(THISAPP) +TARGET = $(DIR_INFO)/$(THISAPP)

+PROG = rsnapshot +PAK_VER = 1

+DEPS = rsync

+SERVICES =

+############################################################################### +# Top-level Rules +###############################################################################

+objects = $(DL_FILE)

+$(DL_FILE) = $(DL_FROM)/$(DL_FILE)

+$(DL_FILE)_BLAKE2 = 2a668aa16991b2b4e611c6204cdcd0e8c9593e5f0af5ea89e787a578e73b6f5987514cd7d0252bb78aea1b157ef85aea947686111ca9e3befdb2a8cef0aa9ecd

+install : $(TARGET)

+check : $(patsubst %,$(DIR_CHK)/%,$(objects))

+download :$(patsubst %,$(DIR_DL)/%,$(objects))

+b2 : $(subst %,%_BLAKE2,$(objects))

+dist:

• @$(PAK)

+############################################################################### +# Downloading, checking, b2sum +###############################################################################

+$(patsubst %,$(DIR_CHK)/%,$(objects)) :

• @$(CHECK)

+$(patsubst %,$(DIR_DL)/%,$(objects)) :

• @$(LOAD)

+$(subst %,%_BLAKE2,$(objects)) :

• @$(B2SUM)

+############################################################################### +# Installation Details +###############################################################################

+$(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects))

• @$(PREBUILD)
• @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar zxf $(DIR_DL)/$(DL_FILE)
• cd $(DIR_APP) && [ -x "configure" ] || sh ./autogen.sh
• cd $(DIR_APP) && ./configure \

• --prefix=/usr		\
  

• --sysconfdir=/etc	
  

• cd $(DIR_APP) && make $(MAKETUNING)
• cd $(DIR_APP) && make install
• # Add conf file to /etc directory
• cp -vf $(DIR_CONF)/rsnapshot/rsnapshot.conf /etc
• # install fcron scripts and backup sqlite script
• install -v -m 755 $(DIR_CONF)/rsnapshot/{rsnapshot-hourly,rsnapshot-daily} \

• $(DIR_CONF)/rsnapshot/{rsnapshot-weekly,rsnapshot-monthly}				\
  

• $(DIR_CONF)/rsnapshot/{rsnapshot-yearly,backup_sqlite.sh}				\
  

• 	-t /var/ipfire/backup/bin
  

• # link rsnapshot interval scripts to various fcron folders
• ln -svf /var/ipfire/backup/bin/rsnapshot-hourly /etc/fcron.hourly
• ln -svf /var/ipfire/backup/bin/rsnapshot-daily /etc/fcron.daily
• ln -svf /var/ipfire/backup/bin/rsnapshot-weekly /etc/fcron.weekly
• ln -svf /var/ipfire/backup/bin/rsnapshot-monthly /etc/fcron.monthly
• #ln -svf /var/ipfire/backup/bin/rsnapshot-yearly /etc/fcron.???
• # Install backup definition
• cp -vf $(DIR_CONF)/backup/includes/rsnapshot /var/ipfire/backup/addons/includes/rsnapshot
• @rm -rf $(DIR_APP)
• @$(POSTBUILD)

diff --git a/make.sh b/make.sh index 3b7f9850c..1b6b1e253 100755 --- a/make.sh +++ b/make.sh @@ -1709,6 +1709,7 @@ buildipfire() { lfsmake2 perl-Imager-QRCode lfsmake2 perl-MIME-Base32 lfsmake2 perl-URI-Encode

• lfsmake2 rsnapshot

}

buildinstaller() {

2.30.2