From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adolf Belka To: development@lists.ipfire.org Subject: Re: [PATCH] backup: Fix broken globbing expansion Date: Tue, 29 Mar 2022 17:23:33 +0200 Message-ID: In-Reply-To: <1AFE3E0B-7BB5-4288-ABE7-FD132F9C11B7@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============7471659746367961356==" List-Id: --===============7471659746367961356== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi All, On 29/03/2022 16:22, Michael Tremer wrote: >=20 >> On 29 Mar 2022, at 15:22, Bernhard Bitsch wrote: >> >> Looks like your include.user file contains absolute paths. That was not the problem but I have corrected that. I would have expected tha= t when the main include and exclude files were made relative that the update = script would also have checked for any entries in the include.user and exclud= e.user files but it looks like it didn't. >=20 > Or an empty line maybe? That turned out to be the problem. There was an empty line after the two line= s I had added. You only saw it if you scrolled the pointer down the file in t= he editor. Removing that blank line removed the line only with a / and the backup then s= uccessfully ran and created a backup file containing all the directories expe= cted. The only thing I found is that there was the following tar error message tar: Exiting with failure status due to previous errors The only thing I found in the backup output was several lines such as tar: /etc/squid/squid.conf.local: Cannot stat: No such file or directory tar: /etc/squid/squid.conf.pre.local: Cannot stat: No such file or directory There were seven lines in total like this and the three I checked were files = that did not exist on my system. I presume that these are files that should b= e backed up if present but don't have to be present so that the error message= s are nothing to worry about. There were no error messages when doing the backup via the WUI. The blank line was only on my vm testbed system but as I clone this for any t= esting activities I will have to check all of my existing vm's to fix this. Glad we resolved this simply. Sorry for the anxiety. Regards, Adolf. >=20 >> >> Am 29.03.2022 um 16:14 schrieb Adolf Belka: >>> Hi All, >>> This is what I get with the pushd/popd version in place on the same system >>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list >>> etc/group >>> etc/hosts >>> etc/hosts.allow >>> etc/hosts.deny >>> etc/httpd/server.crt >>> etc/httpd/server.csr >>> etc/httpd/server-ecdsa.crt >>> etc/httpd/server-ecdsa.csr >>> etc/httpd/server-ecdsa.key >>> etc/httpd/server.key >>> etc/ipsec.user.conf >>> etc/ipsec.user.secrets >>> etc/logrotate.d >>> etc/passwd >>> etc/shadow >>> etc/ssh/sshd_config >>> etc/ssh/ssh_host_ecdsa_key >>> etc/ssh/ssh_host_ecdsa_key.pub >>> etc/ssh/ssh_host_ed25519_key >>> etc/ssh/ssh_host_ed25519_key.pub >>> etc/ssh/ssh_host_rsa_key >>> etc/ssh/ssh_host_rsa_key.pub >>> /etc/sudoers >>> etc/sysconfig/createfiles >>> etc/sysconfig/firewall.local >>> etc/sysconfig/lm_sensors >>> etc/sysconfig/modules >>> etc/sysconfig/ramdisk >>> etc/sysconfig/rc >>> etc/sysconfig/rc.local >>> etc/unbound >>> /home/ahb >>> root/.bash_history >>> var/ipfire/accounting/settings.conf >>> var/ipfire/auth/users >>> var/ipfire/backup/addons/backup >>> var/ipfire/backup/exclude.user >>> var/ipfire/backup/include.user >>> var/ipfire/ca/cacert.pem >>> var/ipfire/captive/agb.txt >>> var/ipfire/captive/clients >>> var/ipfire/captive/coupons >>> var/ipfire/captive/logo.dat >>> var/ipfire/captive/settings >>> var/ipfire/captive/terms.txt >>> var/ipfire/captive/voucher_out >>> var/ipfire/certs/hostcert.pem >>> var/ipfire/certs/hostkey.pem >>> var/ipfire/certs/phoebevmipseccert.pem >>> var/ipfire/connscheduler/connscheduler.conf >>> var/ipfire/crls/cacrl.pem >>> var/ipfire/cups/cups-browsed.conf >>> var/ipfire/cups/subscriptions.conf >>> var/ipfire/ddns/config >>> var/ipfire/ddns/ddns.conf >>> var/ipfire/ddns/settings >>> var/ipfire/dhcp/advoptions >>> var/ipfire/dhcp/advoptions-list >>> var/ipfire/dhcpc/dhcpcd.conf >>> var/ipfire/dhcp/dhcpd.conf >>> var/ipfire/dhcp/dhcpd.conf.local >>> var/ipfire/dhcp/enable_blue >>> var/ipfire/dhcp/enable_green >>> var/ipfire/dhcp/fixleases >>> var/ipfire/dhcp/settings >>> var/ipfire/dma/auth.conf >>> var/ipfire/dma/dma.conf >>> var/ipfire/dma/mail.conf >>> var/ipfire/dns >>> var/ipfire/dnsforward/config >>> var/ipfire/dns/settings >>> var/ipfire/ethernet/aliases >>> var/ipfire/ethernet/settings >>> var/ipfire/ethernet/wireless >>> var/ipfire/extrahd/settings >>> var/ipfire/firewall >>> var/ipfire/firewall/config >>> var/ipfire/firewall/settings >>> var/ipfire/fwhosts >>> var/ipfire/isdn/settings >>> var/ipfire/logging/settings >>> var/ipfire/mac/settings >>> var/ipfire/main/firstsetup_ok >>> var/ipfire/main/gpl_accepted >>> var/ipfire/main/hostname.conf >>> var/ipfire/main/hosts >>> var/ipfire/main/manualpages >>> var/ipfire/main/routing >>> var/ipfire/main/security >>> var/ipfire/main/send_profile >>> var/ipfire/main/settings >>> var/ipfire/modem/settings >>> var/ipfire/optionsfw/settings >>> var/ipfire/ovpn >>> var/ipfire/ovpn/ccd.conf >>> var/ipfire/ovpn/collectd.vpn >>> var/ipfire/ovpn/enable >>> var/ipfire/ovpn/server.conf >>> var/ipfire/ovpn/settings >>> var/ipfire/pakfire/settings >>> var/ipfire/ppp >>> var/ipfire/ppp/fake-resolv.conf >>> var/ipfire/ppp/settings >>> var/ipfire/private/cakey.pem >>> var/ipfire/proxy >>> var/ipfire/proxy/asnbl-helper.conf >>> var/ipfire/proxy/cachemgr.conf >>> var/ipfire/proxy/enable >>> var/ipfire/proxy/settings >>> var/ipfire/proxy/squid.conf >>> var/ipfire/qos/bin >>> var/ipfire/qos/classes >>> var/ipfire/qos/level7config >>> var/ipfire/qos/portconfig >>> var/ipfire/qos/settings >>> var/ipfire/qos/subclasses >>> var/ipfire/qos/tosconfig >>> var/ipfire/remote/enablessh >>> var/ipfire/remote/settings >>> var/ipfire/sensors/settings >>> var/ipfire/suricata/oinkmaster.conf >>> var/ipfire/suricata/oinkmaster-modify-sids.conf >>> var/ipfire/suricata/oinkmaster-provider-includes.conf >>> var/ipfire/suricata/providers-settings >>> var/ipfire/suricata/settings >>> var/ipfire/suricata/suricata-default-rules.yaml >>> var/ipfire/suricata/suricata-dns-servers.yaml >>> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml >>> var/ipfire/suricata/suricata-homenet.yaml >>> var/ipfire/suricata/suricata-http-ports.yaml >>> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml >>> var/ipfire/suricata/suricata-used-providers.yaml >>> var/ipfire/time/ >>> var/ipfire/time/counter.conf >>> var/ipfire/time/enable >>> var/ipfire/time/settime.conf >>> var/ipfire/time/settings >>> var/ipfire/upnp/settings >>> var/ipfire/urlfilter >>> var/ipfire/urlfilter/settings >>> var/ipfire/urlfilter/squidGuard.conf >>> var/ipfire/vpn >>> var/ipfire/vpn/config >>> var/ipfire/vpn/ipsec.conf >>> var/ipfire/vpn/settings >>> var/ipfire/wakeonlan/clients.conf >>> var/ipfire/wio/wio.conf >>> var/ipfire/wireless/config >>> var/ipfire/wireless/settings >>> var/lib/suricata >>> var/log/rrd/collectd >>> var/log/rrd/hddshutdown-md127.rrd >>> var/log/rrd/hddshutdown-sda.rrd >>> var/log/rrd/hddshutdown-sdb.rrd >>> var/log/rrd/hddtemp-md127.rrd >>> var/log/rrd/hddtemp-sda.rrd >>> var/log/rrd/hddtemp-sdb.rrd >>> var/log/rrd/wio >>> var/log/vnstat >>> var/tmp/idsrules-emerging.tar.gz >>> var/tmp/idsrules-sslbl_blacklist.rules >>> The following are in the previous list but not in this one:- >>> / >>> /etc/conntrackd/conntrackd.conf >>> /etc/ipsec.user-post.conf >>> /root/.gitconfig >>> /root/.ssh >>> Regards, >>> Adolf. >>> On 29/03/2022 16:01, Adolf Belka wrote: >>>> Hi All, >>>> >>>> On 29/03/2022 15:36, Bernhard Bitsch wrote: >>>>> Hi, >>>>> >>>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer: >>>>>> Hello, >>>>>> >>>>>> Could you please send the file listing to find out what is being inclu= ded what shouldn=E2=80=99t? >>>> Unfortunately, as I stopped the backup continuing once it had reached 1.= 2GB, the file created was not able to be opened. Probably stopping the backup= corrupted it in some way. >>>>>> >>>>> >>>>> Just for easy quick handling, you can use >>>>> '/var/ipfire/backup/bin/backup.pl list' to show the files included. >>>> I gave this a go and here is the output from the command. >>>> >>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list >>>> / >>>> /etc/conntrackd/conntrackd.conf >>>> /etc/group >>>> /etc/hosts >>>> /etc/hosts.allow >>>> /etc/hosts.deny >>>> /etc/httpd/server.crt >>>> /etc/httpd/server.csr >>>> /etc/httpd/server-ecdsa.crt >>>> /etc/httpd/server-ecdsa.csr >>>> /etc/httpd/server-ecdsa.key >>>> /etc/httpd/server.key >>>> /etc/ipsec.user.conf >>>> /etc/ipsec.user-post.conf >>>> /etc/ipsec.user.secrets >>>> /etc/logrotate.d >>>> /etc/passwd >>>> /etc/shadow >>>> /etc/squid/squid.conf.local >>>> /etc/squid/squid.conf.pre.local >>>> /etc/ssh/sshd_config >>>> /etc/ssh/ssh_host_ecdsa_key >>>> /etc/ssh/ssh_host_ecdsa_key.pub >>>> /etc/ssh/ssh_host_ed25519_key >>>> /etc/ssh/ssh_host_ed25519_key.pub >>>> /etc/ssh/ssh_host_rsa_key >>>> /etc/ssh/ssh_host_rsa_key.pub >>>> //etc/sudoers >>>> /etc/sysconfig/createfiles >>>> /etc/sysconfig/firewall.local >>>> /etc/sysconfig/lm_sensors >>>> /etc/sysconfig/modules >>>> /etc/sysconfig/ramdisk >>>> /etc/sysconfig/rc >>>> /etc/sysconfig/rc.local >>>> /etc/unbound >>>> //home/ahb >>>> /root/.bash_history >>>> /root/.gitconfig >>>> /root/.ssh >>>> /var/ipfire/accounting/settings.conf >>>> /var/ipfire/auth/users >>>> /var/ipfire/backup/addons/backup >>>> /var/ipfire/backup/exclude.user >>>> /var/ipfire/backup/include.user >>>> /var/ipfire/ca/cacert.pem >>>> /var/ipfire/captive/agb.txt >>>> /var/ipfire/captive/clients >>>> /var/ipfire/captive/coupons >>>> /var/ipfire/captive/logo.dat >>>> /var/ipfire/captive/settings >>>> /var/ipfire/captive/terms.txt >>>> /var/ipfire/captive/voucher_out >>>> /var/ipfire/certs/hostcert.pem >>>> /var/ipfire/certs/hostkey.pem >>>> /var/ipfire/certs/phoebevmipseccert.pem >>>> /var/ipfire/connscheduler/connscheduler.conf >>>> /var/ipfire/crls/cacrl.pem >>>> /var/ipfire/cups/cups-browsed.conf >>>> /var/ipfire/cups/subscriptions.conf >>>> /var/ipfire/ddns/config >>>> /var/ipfire/ddns/ddns.conf >>>> /var/ipfire/ddns/settings >>>> /var/ipfire/dhcp/advoptions >>>> /var/ipfire/dhcp/advoptions-list >>>> /var/ipfire/dhcpc/dhcpcd.conf >>>> /var/ipfire/dhcp/dhcpd.conf >>>> /var/ipfire/dhcp/dhcpd.conf.local >>>> /var/ipfire/dhcp/enable_blue >>>> /var/ipfire/dhcp/enable_green >>>> /var/ipfire/dhcp/fixleases >>>> /var/ipfire/dhcp/settings >>>> /var/ipfire/dma/auth.conf >>>> /var/ipfire/dma/dma.conf >>>> /var/ipfire/dma/mail.conf >>>> /var/ipfire/dns >>>> /var/ipfire/dnsforward/config >>>> /var/ipfire/dns/settings >>>> /var/ipfire/ethernet/aliases >>>> /var/ipfire/ethernet/settings >>>> /var/ipfire/ethernet/wireless >>>> /var/ipfire/extrahd/settings >>>> /var/ipfire/firewall >>>> /var/ipfire/firewall/config >>>> /var/ipfire/firewall/settings >>>> /var/ipfire/fwhosts >>>> /var/ipfire/isdn/settings >>>> /var/ipfire/logging/settings >>>> /var/ipfire/mac/settings >>>> /var/ipfire/main/firstsetup_ok >>>> /var/ipfire/main/gpl_accepted >>>> /var/ipfire/main/hostname.conf >>>> /var/ipfire/main/hosts >>>> /var/ipfire/main/manualpages >>>> /var/ipfire/main/routing >>>> /var/ipfire/main/security >>>> /var/ipfire/main/send_profile >>>> /var/ipfire/main/settings >>>> /var/ipfire/modem/settings >>>> /var/ipfire/optionsfw/settings >>>> /var/ipfire/ovpn >>>> /var/ipfire/ovpn/ccd.conf >>>> /var/ipfire/ovpn/collectd.vpn >>>> /var/ipfire/ovpn/enable >>>> /var/ipfire/ovpn/server.conf >>>> /var/ipfire/ovpn/settings >>>> /var/ipfire/pakfire/settings >>>> /var/ipfire/ppp >>>> /var/ipfire/ppp/fake-resolv.conf >>>> /var/ipfire/ppp/settings >>>> /var/ipfire/private/cakey.pem >>>> /var/ipfire/proxy >>>> /var/ipfire/proxy/asnbl-helper.conf >>>> /var/ipfire/proxy/cachemgr.conf >>>> /var/ipfire/proxy/enable >>>> /var/ipfire/proxy/settings >>>> /var/ipfire/proxy/squid.conf >>>> /var/ipfire/qos/bin >>>> /var/ipfire/qos/bin/qos.sh >>>> /var/ipfire/qos/classes >>>> /var/ipfire/qos/level7config >>>> /var/ipfire/qos/portconfig >>>> /var/ipfire/qos/settings >>>> /var/ipfire/qos/subclasses >>>> /var/ipfire/qos/tosconfig >>>> /var/ipfire/remote/enablessh >>>> /var/ipfire/remote/settings >>>> /var/ipfire/sensors/settings >>>> /var/ipfire/suricata/oinkmaster.conf >>>> /var/ipfire/suricata/oinkmaster-modify-sids.conf >>>> /var/ipfire/suricata/oinkmaster-provider-includes.conf >>>> /var/ipfire/suricata/providers-settings >>>> /var/ipfire/suricata/settings >>>> /var/ipfire/suricata/suricata-default-rules.yaml >>>> /var/ipfire/suricata/suricata-dns-servers.yaml >>>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml >>>> /var/ipfire/suricata/suricata-homenet.yaml >>>> /var/ipfire/suricata/suricata-http-ports.yaml >>>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml >>>> /var/ipfire/suricata/suricata-used-providers.yaml >>>> /var/ipfire/time/ >>>> /var/ipfire/time/counter.conf >>>> /var/ipfire/time/enable >>>> /var/ipfire/time/settime.conf >>>> /var/ipfire/time/settings >>>> /var/ipfire/upnp/settings >>>> /var/ipfire/urlfilter >>>> /var/ipfire/urlfilter/settings >>>> /var/ipfire/urlfilter/squidGuard.conf >>>> /var/ipfire/vpn >>>> /var/ipfire/vpn/config >>>> /var/ipfire/vpn/ipsec.conf >>>> /var/ipfire/vpn/settings >>>> /var/ipfire/wakeonlan/clients.conf >>>> /var/ipfire/wio/wio.conf >>>> /var/ipfire/wireless/config >>>> /var/ipfire/wireless/settings >>>> /var/lib/suricata >>>> /var/log/rrd/collectd >>>> /var/log/rrd/hddshutdown-md127.rrd >>>> /var/log/rrd/hddshutdown-sda.rrd >>>> /var/log/rrd/hddshutdown-sdb.rrd >>>> /var/log/rrd/hddtemp-md127.rrd >>>> /var/log/rrd/hddtemp-sda.rrd >>>> /var/log/rrd/hddtemp-sdb.rrd >>>> /var/log/rrd/wio >>>> /var/log/vnstat >>>> /var/tmp/idsrules-emerging.tar.gz >>>> /var/tmp/idsrules-sslbl_blacklist.rules >>>> >>>> Regards, >>>> Adolf. >>>>> >>>>> Bernhard >>>>>> -Michael >>>>>> >>>>>>> On 29 Mar 2022, at 14:10, Adolf Belka wrot= e: >>>>>>> >>>>>>> Hi Michael, >>>>>>> >>>>>>> Just tried this patch out on my vm testbed system and it still doesn'= t work for me. The backup file had got to 1.3GB when I deleted the backup fil= e as it was still growing. The normal correct backup file on that vm machine = is around 7MB >>>>>>> >>>>>>> The above was the case for both running it from the WUI or from the c= ommand line from my unprivileged user using sudo backupctrl exclude >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Adolf. >>>>>>> >>>>>>> >>>>>>> On 29/03/2022 14:27, Michael Tremer wrote: >>>>>>>> This patch fixes globbing expansion in the backup include file list >>>>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b. >>>>>>>> >>>>>>>> Signed-off-by: Michael Tremer >>>>>>>> --- >>>>>>>> config/backup/backup.pl | 8 ++++---- >>>>>>>> 1 file changed, 4 insertions(+), 4 deletions(-) >>>>>>>> >>>>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl >>>>>>>> index a2337cf23..6f9295e94 100644 >>>>>>>> --- a/config/backup/backup.pl >>>>>>>> +++ b/config/backup/backup.pl >>>>>>>> @@ -19,6 +19,8 @@ >>>>>>>> # # >>>>>>>> ####################################################################= ########### >>>>>>>> +shopt -s nullglob >>>>>>>> + >>>>>>>> NOW=3D"$(date "+%Y-%m-%d-%H:%M")" >>>>>>>> list_addons() { >>>>>>>> @@ -38,10 +40,8 @@ process_includes() { >>>>>>>> for include in $@; do >>>>>>>> local file >>>>>>>> while read -r file; do >>>>>>>> - for file in ${file}; do >>>>>>>> - if [ -e "/${file}" ]; then >>>>>>>> - echo "${file}" >>>>>>>> - fi >>>>>>>> + for file in /${file}; do >>>>>>>> + echo "${file}" >>>>>>>> done >>>>>>>> done < "${include}" >>>>>>>> done | sort -u >>>>>> >=20 --===============7471659746367961356==--