Re: [PATCH] backup: Fix broken globbing expansion

[Adolf Belka <adolf.belka@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 15908 bytes --]

Hi All,

On 29/03/2022 16:22, Michael Tremer wrote:
> 
>> On 29 Mar 2022, at 15:22, Bernhard Bitsch <bbitsch(a)ipfire.org> wrote:
>>
>> Looks like your include.user file contains absolute paths.
That was not the problem but I have corrected that. I would have expected that when the main include and exclude files were made relative that the update script would also have checked for any entries in the include.user and exclude.user files but it looks like it didn't.
> 
> Or an empty line maybe?
That turned out to be the problem. There was an empty line after the two lines I had added. You only saw it if you scrolled the pointer down the file in the editor.

Removing that blank line removed the line only with a / and the backup then successfully ran and created a backup file containing all the directories expected.

The only thing I found is that there was the following tar error message

tar: Exiting with failure status due to previous errors

The only thing I found in the backup output was several lines such as

tar: /etc/squid/squid.conf.local: Cannot stat: No such file or directory
tar: /etc/squid/squid.conf.pre.local: Cannot stat: No such file or directory

There were seven lines in total like this and the three I checked were files that did not exist on my system. I presume that these are files that should be backed up if present but don't have to be present so that the error messages are nothing to worry about.
There were no error messages when doing the backup via the WUI.

The blank line was only on my vm testbed system but as I clone this for any testing activities I will have to check all of my existing vm's to fix this.

Glad we resolved this simply. Sorry for the anxiety.

Regards,

Adolf.

> 
>>
>> Am 29.03.2022 um 16:14 schrieb Adolf Belka:
>>> Hi All,
>>> This is what I get with the pushd/popd version in place on the same system
>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>> etc/group
>>> etc/hosts
>>> etc/hosts.allow
>>> etc/hosts.deny
>>> etc/httpd/server.crt
>>> etc/httpd/server.csr
>>> etc/httpd/server-ecdsa.crt
>>> etc/httpd/server-ecdsa.csr
>>> etc/httpd/server-ecdsa.key
>>> etc/httpd/server.key
>>> etc/ipsec.user.conf
>>> etc/ipsec.user.secrets
>>> etc/logrotate.d
>>> etc/passwd
>>> etc/shadow
>>> etc/ssh/sshd_config
>>> etc/ssh/ssh_host_ecdsa_key
>>> etc/ssh/ssh_host_ecdsa_key.pub
>>> etc/ssh/ssh_host_ed25519_key
>>> etc/ssh/ssh_host_ed25519_key.pub
>>> etc/ssh/ssh_host_rsa_key
>>> etc/ssh/ssh_host_rsa_key.pub
>>> /etc/sudoers
>>> etc/sysconfig/createfiles
>>> etc/sysconfig/firewall.local
>>> etc/sysconfig/lm_sensors
>>> etc/sysconfig/modules
>>> etc/sysconfig/ramdisk
>>> etc/sysconfig/rc
>>> etc/sysconfig/rc.local
>>> etc/unbound
>>> /home/ahb
>>> root/.bash_history
>>> var/ipfire/accounting/settings.conf
>>> var/ipfire/auth/users
>>> var/ipfire/backup/addons/backup
>>> var/ipfire/backup/exclude.user
>>> var/ipfire/backup/include.user
>>> var/ipfire/ca/cacert.pem
>>> var/ipfire/captive/agb.txt
>>> var/ipfire/captive/clients
>>> var/ipfire/captive/coupons
>>> var/ipfire/captive/logo.dat
>>> var/ipfire/captive/settings
>>> var/ipfire/captive/terms.txt
>>> var/ipfire/captive/voucher_out
>>> var/ipfire/certs/hostcert.pem
>>> var/ipfire/certs/hostkey.pem
>>> var/ipfire/certs/phoebevmipseccert.pem
>>> var/ipfire/connscheduler/connscheduler.conf
>>> var/ipfire/crls/cacrl.pem
>>> var/ipfire/cups/cups-browsed.conf
>>> var/ipfire/cups/subscriptions.conf
>>> var/ipfire/ddns/config
>>> var/ipfire/ddns/ddns.conf
>>> var/ipfire/ddns/settings
>>> var/ipfire/dhcp/advoptions
>>> var/ipfire/dhcp/advoptions-list
>>> var/ipfire/dhcpc/dhcpcd.conf
>>> var/ipfire/dhcp/dhcpd.conf
>>> var/ipfire/dhcp/dhcpd.conf.local
>>> var/ipfire/dhcp/enable_blue
>>> var/ipfire/dhcp/enable_green
>>> var/ipfire/dhcp/fixleases
>>> var/ipfire/dhcp/settings
>>> var/ipfire/dma/auth.conf
>>> var/ipfire/dma/dma.conf
>>> var/ipfire/dma/mail.conf
>>> var/ipfire/dns
>>> var/ipfire/dnsforward/config
>>> var/ipfire/dns/settings
>>> var/ipfire/ethernet/aliases
>>> var/ipfire/ethernet/settings
>>> var/ipfire/ethernet/wireless
>>> var/ipfire/extrahd/settings
>>> var/ipfire/firewall
>>> var/ipfire/firewall/config
>>> var/ipfire/firewall/settings
>>> var/ipfire/fwhosts
>>> var/ipfire/isdn/settings
>>> var/ipfire/logging/settings
>>> var/ipfire/mac/settings
>>> var/ipfire/main/firstsetup_ok
>>> var/ipfire/main/gpl_accepted
>>> var/ipfire/main/hostname.conf
>>> var/ipfire/main/hosts
>>> var/ipfire/main/manualpages
>>> var/ipfire/main/routing
>>> var/ipfire/main/security
>>> var/ipfire/main/send_profile
>>> var/ipfire/main/settings
>>> var/ipfire/modem/settings
>>> var/ipfire/optionsfw/settings
>>> var/ipfire/ovpn
>>> var/ipfire/ovpn/ccd.conf
>>> var/ipfire/ovpn/collectd.vpn
>>> var/ipfire/ovpn/enable
>>> var/ipfire/ovpn/server.conf
>>> var/ipfire/ovpn/settings
>>> var/ipfire/pakfire/settings
>>> var/ipfire/ppp
>>> var/ipfire/ppp/fake-resolv.conf
>>> var/ipfire/ppp/settings
>>> var/ipfire/private/cakey.pem
>>> var/ipfire/proxy
>>> var/ipfire/proxy/asnbl-helper.conf
>>> var/ipfire/proxy/cachemgr.conf
>>> var/ipfire/proxy/enable
>>> var/ipfire/proxy/settings
>>> var/ipfire/proxy/squid.conf
>>> var/ipfire/qos/bin
>>> var/ipfire/qos/classes
>>> var/ipfire/qos/level7config
>>> var/ipfire/qos/portconfig
>>> var/ipfire/qos/settings
>>> var/ipfire/qos/subclasses
>>> var/ipfire/qos/tosconfig
>>> var/ipfire/remote/enablessh
>>> var/ipfire/remote/settings
>>> var/ipfire/sensors/settings
>>> var/ipfire/suricata/oinkmaster.conf
>>> var/ipfire/suricata/oinkmaster-modify-sids.conf
>>> var/ipfire/suricata/oinkmaster-provider-includes.conf
>>> var/ipfire/suricata/providers-settings
>>> var/ipfire/suricata/settings
>>> var/ipfire/suricata/suricata-default-rules.yaml
>>> var/ipfire/suricata/suricata-dns-servers.yaml
>>> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>> var/ipfire/suricata/suricata-homenet.yaml
>>> var/ipfire/suricata/suricata-http-ports.yaml
>>> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>> var/ipfire/suricata/suricata-used-providers.yaml
>>> var/ipfire/time/
>>> var/ipfire/time/counter.conf
>>> var/ipfire/time/enable
>>> var/ipfire/time/settime.conf
>>> var/ipfire/time/settings
>>> var/ipfire/upnp/settings
>>> var/ipfire/urlfilter
>>> var/ipfire/urlfilter/settings
>>> var/ipfire/urlfilter/squidGuard.conf
>>> var/ipfire/vpn
>>> var/ipfire/vpn/config
>>> var/ipfire/vpn/ipsec.conf
>>> var/ipfire/vpn/settings
>>> var/ipfire/wakeonlan/clients.conf
>>> var/ipfire/wio/wio.conf
>>> var/ipfire/wireless/config
>>> var/ipfire/wireless/settings
>>> var/lib/suricata
>>> var/log/rrd/collectd
>>> var/log/rrd/hddshutdown-md127.rrd
>>> var/log/rrd/hddshutdown-sda.rrd
>>> var/log/rrd/hddshutdown-sdb.rrd
>>> var/log/rrd/hddtemp-md127.rrd
>>> var/log/rrd/hddtemp-sda.rrd
>>> var/log/rrd/hddtemp-sdb.rrd
>>> var/log/rrd/wio
>>> var/log/vnstat
>>> var/tmp/idsrules-emerging.tar.gz
>>> var/tmp/idsrules-sslbl_blacklist.rules
>>> The following are in the previous list but not in this one:-
>>> /
>>> /etc/conntrackd/conntrackd.conf
>>> /etc/ipsec.user-post.conf
>>> /root/.gitconfig
>>> /root/.ssh
>>> Regards,
>>> Adolf.
>>> On 29/03/2022 16:01, Adolf Belka wrote:
>>>> Hi All,
>>>>
>>>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>>>> Hi,
>>>>>
>>>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>>>> Hello,
>>>>>>
>>>>>> Could you please send the file listing to find out what is being included what shouldn’t?
>>>> Unfortunately, as I stopped the backup continuing once it had reached 1.2GB, the file created was not able to be opened. Probably stopping the backup corrupted it in some way.
>>>>>>
>>>>>
>>>>> Just for easy quick handling, you can use
>>>>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
>>>> I gave this a go and here is the output from the command.
>>>>
>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>> /
>>>> /etc/conntrackd/conntrackd.conf
>>>> /etc/group
>>>> /etc/hosts
>>>> /etc/hosts.allow
>>>> /etc/hosts.deny
>>>> /etc/httpd/server.crt
>>>> /etc/httpd/server.csr
>>>> /etc/httpd/server-ecdsa.crt
>>>> /etc/httpd/server-ecdsa.csr
>>>> /etc/httpd/server-ecdsa.key
>>>> /etc/httpd/server.key
>>>> /etc/ipsec.user.conf
>>>> /etc/ipsec.user-post.conf
>>>> /etc/ipsec.user.secrets
>>>> /etc/logrotate.d
>>>> /etc/passwd
>>>> /etc/shadow
>>>> /etc/squid/squid.conf.local
>>>> /etc/squid/squid.conf.pre.local
>>>> /etc/ssh/sshd_config
>>>> /etc/ssh/ssh_host_ecdsa_key
>>>> /etc/ssh/ssh_host_ecdsa_key.pub
>>>> /etc/ssh/ssh_host_ed25519_key
>>>> /etc/ssh/ssh_host_ed25519_key.pub
>>>> /etc/ssh/ssh_host_rsa_key
>>>> /etc/ssh/ssh_host_rsa_key.pub
>>>> //etc/sudoers
>>>> /etc/sysconfig/createfiles
>>>> /etc/sysconfig/firewall.local
>>>> /etc/sysconfig/lm_sensors
>>>> /etc/sysconfig/modules
>>>> /etc/sysconfig/ramdisk
>>>> /etc/sysconfig/rc
>>>> /etc/sysconfig/rc.local
>>>> /etc/unbound
>>>> //home/ahb
>>>> /root/.bash_history
>>>> /root/.gitconfig
>>>> /root/.ssh
>>>> /var/ipfire/accounting/settings.conf
>>>> /var/ipfire/auth/users
>>>> /var/ipfire/backup/addons/backup
>>>> /var/ipfire/backup/exclude.user
>>>> /var/ipfire/backup/include.user
>>>> /var/ipfire/ca/cacert.pem
>>>> /var/ipfire/captive/agb.txt
>>>> /var/ipfire/captive/clients
>>>> /var/ipfire/captive/coupons
>>>> /var/ipfire/captive/logo.dat
>>>> /var/ipfire/captive/settings
>>>> /var/ipfire/captive/terms.txt
>>>> /var/ipfire/captive/voucher_out
>>>> /var/ipfire/certs/hostcert.pem
>>>> /var/ipfire/certs/hostkey.pem
>>>> /var/ipfire/certs/phoebevmipseccert.pem
>>>> /var/ipfire/connscheduler/connscheduler.conf
>>>> /var/ipfire/crls/cacrl.pem
>>>> /var/ipfire/cups/cups-browsed.conf
>>>> /var/ipfire/cups/subscriptions.conf
>>>> /var/ipfire/ddns/config
>>>> /var/ipfire/ddns/ddns.conf
>>>> /var/ipfire/ddns/settings
>>>> /var/ipfire/dhcp/advoptions
>>>> /var/ipfire/dhcp/advoptions-list
>>>> /var/ipfire/dhcpc/dhcpcd.conf
>>>> /var/ipfire/dhcp/dhcpd.conf
>>>> /var/ipfire/dhcp/dhcpd.conf.local
>>>> /var/ipfire/dhcp/enable_blue
>>>> /var/ipfire/dhcp/enable_green
>>>> /var/ipfire/dhcp/fixleases
>>>> /var/ipfire/dhcp/settings
>>>> /var/ipfire/dma/auth.conf
>>>> /var/ipfire/dma/dma.conf
>>>> /var/ipfire/dma/mail.conf
>>>> /var/ipfire/dns
>>>> /var/ipfire/dnsforward/config
>>>> /var/ipfire/dns/settings
>>>> /var/ipfire/ethernet/aliases
>>>> /var/ipfire/ethernet/settings
>>>> /var/ipfire/ethernet/wireless
>>>> /var/ipfire/extrahd/settings
>>>> /var/ipfire/firewall
>>>> /var/ipfire/firewall/config
>>>> /var/ipfire/firewall/settings
>>>> /var/ipfire/fwhosts
>>>> /var/ipfire/isdn/settings
>>>> /var/ipfire/logging/settings
>>>> /var/ipfire/mac/settings
>>>> /var/ipfire/main/firstsetup_ok
>>>> /var/ipfire/main/gpl_accepted
>>>> /var/ipfire/main/hostname.conf
>>>> /var/ipfire/main/hosts
>>>> /var/ipfire/main/manualpages
>>>> /var/ipfire/main/routing
>>>> /var/ipfire/main/security
>>>> /var/ipfire/main/send_profile
>>>> /var/ipfire/main/settings
>>>> /var/ipfire/modem/settings
>>>> /var/ipfire/optionsfw/settings
>>>> /var/ipfire/ovpn
>>>> /var/ipfire/ovpn/ccd.conf
>>>> /var/ipfire/ovpn/collectd.vpn
>>>> /var/ipfire/ovpn/enable
>>>> /var/ipfire/ovpn/server.conf
>>>> /var/ipfire/ovpn/settings
>>>> /var/ipfire/pakfire/settings
>>>> /var/ipfire/ppp
>>>> /var/ipfire/ppp/fake-resolv.conf
>>>> /var/ipfire/ppp/settings
>>>> /var/ipfire/private/cakey.pem
>>>> /var/ipfire/proxy
>>>> /var/ipfire/proxy/asnbl-helper.conf
>>>> /var/ipfire/proxy/cachemgr.conf
>>>> /var/ipfire/proxy/enable
>>>> /var/ipfire/proxy/settings
>>>> /var/ipfire/proxy/squid.conf
>>>> /var/ipfire/qos/bin
>>>> /var/ipfire/qos/bin/qos.sh
>>>> /var/ipfire/qos/classes
>>>> /var/ipfire/qos/level7config
>>>> /var/ipfire/qos/portconfig
>>>> /var/ipfire/qos/settings
>>>> /var/ipfire/qos/subclasses
>>>> /var/ipfire/qos/tosconfig
>>>> /var/ipfire/remote/enablessh
>>>> /var/ipfire/remote/settings
>>>> /var/ipfire/sensors/settings
>>>> /var/ipfire/suricata/oinkmaster.conf
>>>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>> /var/ipfire/suricata/providers-settings
>>>> /var/ipfire/suricata/settings
>>>> /var/ipfire/suricata/suricata-default-rules.yaml
>>>> /var/ipfire/suricata/suricata-dns-servers.yaml
>>>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>> /var/ipfire/suricata/suricata-homenet.yaml
>>>> /var/ipfire/suricata/suricata-http-ports.yaml
>>>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>> /var/ipfire/suricata/suricata-used-providers.yaml
>>>> /var/ipfire/time/
>>>> /var/ipfire/time/counter.conf
>>>> /var/ipfire/time/enable
>>>> /var/ipfire/time/settime.conf
>>>> /var/ipfire/time/settings
>>>> /var/ipfire/upnp/settings
>>>> /var/ipfire/urlfilter
>>>> /var/ipfire/urlfilter/settings
>>>> /var/ipfire/urlfilter/squidGuard.conf
>>>> /var/ipfire/vpn
>>>> /var/ipfire/vpn/config
>>>> /var/ipfire/vpn/ipsec.conf
>>>> /var/ipfire/vpn/settings
>>>> /var/ipfire/wakeonlan/clients.conf
>>>> /var/ipfire/wio/wio.conf
>>>> /var/ipfire/wireless/config
>>>> /var/ipfire/wireless/settings
>>>> /var/lib/suricata
>>>> /var/log/rrd/collectd
>>>> /var/log/rrd/hddshutdown-md127.rrd
>>>> /var/log/rrd/hddshutdown-sda.rrd
>>>> /var/log/rrd/hddshutdown-sdb.rrd
>>>> /var/log/rrd/hddtemp-md127.rrd
>>>> /var/log/rrd/hddtemp-sda.rrd
>>>> /var/log/rrd/hddtemp-sdb.rrd
>>>> /var/log/rrd/wio
>>>> /var/log/vnstat
>>>> /var/tmp/idsrules-emerging.tar.gz
>>>> /var/tmp/idsrules-sslbl_blacklist.rules
>>>>
>>>> Regards,
>>>> Adolf.
>>>>>
>>>>> Bernhard
>>>>>> -Michael
>>>>>>
>>>>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>
>>>>>>> Hi Michael,
>>>>>>>
>>>>>>> Just tried this patch out on my vm testbed system and it still doesn't work for me. The backup file had got to 1.3GB when I deleted the backup file as it was still growing. The normal correct backup file on that vm machine is around 7MB
>>>>>>>
>>>>>>> The above was the case for both running it from the WUI or from the command line from my unprivileged user using sudo backupctrl exclude
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Adolf.
>>>>>>>
>>>>>>>
>>>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>>>> This patch fixes globbing expansion in the backup include file list
>>>>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>>>
>>>>>>>> Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org>
>>>>>>>> ---
>>>>>>>>    config/backup/backup.pl | 8 ++++----
>>>>>>>>    1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>>>>
>>>>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>>>>>> index a2337cf23..6f9295e94 100644
>>>>>>>> --- a/config/backup/backup.pl
>>>>>>>> +++ b/config/backup/backup.pl
>>>>>>>> @@ -19,6 +19,8 @@
>>>>>>>> # #
>>>>>>>> ###############################################################################
>>>>>>>>    +shopt -s nullglob
>>>>>>>> +
>>>>>>>>    NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>>>>      list_addons() {
>>>>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>>>>        for include in $@; do
>>>>>>>>            local file
>>>>>>>>            while read -r file; do
>>>>>>>> -            for file in ${file}; do
>>>>>>>> -                if [ -e "/${file}" ]; then
>>>>>>>> -                    echo "${file}"
>>>>>>>> -                fi
>>>>>>>> +            for file in /${file}; do
>>>>>>>> +                echo "${file}"
>>>>>>>>                done
>>>>>>>>            done < "${include}"
>>>>>>>>        done | sort -u
>>>>>>
>