Re: [PATCH] backup: Fix broken globbing expansion

[Bernhard Bitsch <bbitsch@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 17245 bytes --]

Hi all,

Am 29.03.2022 um 17:23 schrieb Adolf Belka:
> Hi All,
> 
> On 29/03/2022 16:22, Michael Tremer wrote:
>>
>>> On 29 Mar 2022, at 15:22, Bernhard Bitsch <bbitsch(a)ipfire.org> wrote:
>>>
>>> Looks like your include.user file contains absolute paths.
> That was not the problem but I have corrected that. I would have 
> expected that when the main include and exclude files were made relative 
> that the update script would also have checked for any entries in the 
> include.user and exclude.user files but it looks like it didn't.th
According to the source they are chosen.
>>
>> Or an empty line maybe?
> That turned out to be the problem. There was an empty line after the two 
> lines I had added. You only saw it if you scrolled the pointer down the 
> file in the editor.
Interesting. A blank line should produce nothing.
> 
> Removing that blank line removed the line only with a / and the backup 
> then successfully ran and created a backup file containing all the 
> directories expected.
> 
> The only thing I found is that there was the following tar error message
> 
> tar: Exiting with failure status due to previous errors
> 
> The only thing I found in the backup output was several lines such as
> 
> tar: /etc/squid/squid.conf.local: Cannot stat: No such file or directory
> tar: /etc/squid/squid.conf.pre.local: Cannot stat: No such file or 
> directory
> 
> There were seven lines in total like this and the three I checked were 
> files that did not exist on my system. I presume that these are files 
> that should be backed up if present but don't have to be present so that 
> the error messages are nothing to worry about.
> There were no error messages when doing the backup via the WUI.
> 

Good find. ;)
Michael's patch removes the test of existence, globbing produces 
existent file names only. But I suppose there are several full names of 
files, not existing in each implementation.
Conclusion: Repair the file globbing by the shopt and do the existence 
check. So all existent files are chosen, that are listed in the include 
files.

Regards,
Bernhard

> The blank line was only on my vm testbed system but as I clone this for 
> any testing activities I will have to check all of my existing vm's to 
> fix this.
> 
> Glad we resolved this simply. Sorry for the anxiety.
> 
> Regards,
> 
> Adolf.
> 
>>
>>>
>>> Am 29.03.2022 um 16:14 schrieb Adolf Belka:
>>>> Hi All,
>>>> This is what I get with the pushd/popd version in place on the same 
>>>> system
>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>> etc/group
>>>> etc/hosts
>>>> etc/hosts.allow
>>>> etc/hosts.deny
>>>> etc/httpd/server.crt
>>>> etc/httpd/server.csr
>>>> etc/httpd/server-ecdsa.crt
>>>> etc/httpd/server-ecdsa.csr
>>>> etc/httpd/server-ecdsa.key
>>>> etc/httpd/server.key
>>>> etc/ipsec.user.conf
>>>> etc/ipsec.user.secrets
>>>> etc/logrotate.d
>>>> etc/passwd
>>>> etc/shadow
>>>> etc/ssh/sshd_config
>>>> etc/ssh/ssh_host_ecdsa_key
>>>> etc/ssh/ssh_host_ecdsa_key.pub
>>>> etc/ssh/ssh_host_ed25519_key
>>>> etc/ssh/ssh_host_ed25519_key.pub
>>>> etc/ssh/ssh_host_rsa_key
>>>> etc/ssh/ssh_host_rsa_key.pub
>>>> /etc/sudoers
>>>> etc/sysconfig/createfiles
>>>> etc/sysconfig/firewall.local
>>>> etc/sysconfig/lm_sensors
>>>> etc/sysconfig/modules
>>>> etc/sysconfig/ramdisk
>>>> etc/sysconfig/rc
>>>> etc/sysconfig/rc.local
>>>> etc/unbound
>>>> /home/ahb
>>>> root/.bash_history
>>>> var/ipfire/accounting/settings.conf
>>>> var/ipfire/auth/users
>>>> var/ipfire/backup/addons/backup
>>>> var/ipfire/backup/exclude.user
>>>> var/ipfire/backup/include.user
>>>> var/ipfire/ca/cacert.pem
>>>> var/ipfire/captive/agb.txt
>>>> var/ipfire/captive/clients
>>>> var/ipfire/captive/coupons
>>>> var/ipfire/captive/logo.dat
>>>> var/ipfire/captive/settings
>>>> var/ipfire/captive/terms.txt
>>>> var/ipfire/captive/voucher_out
>>>> var/ipfire/certs/hostcert.pem
>>>> var/ipfire/certs/hostkey.pem
>>>> var/ipfire/certs/phoebevmipseccert.pem
>>>> var/ipfire/connscheduler/connscheduler.conf
>>>> var/ipfire/crls/cacrl.pem
>>>> var/ipfire/cups/cups-browsed.conf
>>>> var/ipfire/cups/subscriptions.conf
>>>> var/ipfire/ddns/config
>>>> var/ipfire/ddns/ddns.conf
>>>> var/ipfire/ddns/settings
>>>> var/ipfire/dhcp/advoptions
>>>> var/ipfire/dhcp/advoptions-list
>>>> var/ipfire/dhcpc/dhcpcd.conf
>>>> var/ipfire/dhcp/dhcpd.conf
>>>> var/ipfire/dhcp/dhcpd.conf.local
>>>> var/ipfire/dhcp/enable_blue
>>>> var/ipfire/dhcp/enable_green
>>>> var/ipfire/dhcp/fixleases
>>>> var/ipfire/dhcp/settings
>>>> var/ipfire/dma/auth.conf
>>>> var/ipfire/dma/dma.conf
>>>> var/ipfire/dma/mail.conf
>>>> var/ipfire/dns
>>>> var/ipfire/dnsforward/config
>>>> var/ipfire/dns/settings
>>>> var/ipfire/ethernet/aliases
>>>> var/ipfire/ethernet/settings
>>>> var/ipfire/ethernet/wireless
>>>> var/ipfire/extrahd/settings
>>>> var/ipfire/firewall
>>>> var/ipfire/firewall/config
>>>> var/ipfire/firewall/settings
>>>> var/ipfire/fwhosts
>>>> var/ipfire/isdn/settings
>>>> var/ipfire/logging/settings
>>>> var/ipfire/mac/settings
>>>> var/ipfire/main/firstsetup_ok
>>>> var/ipfire/main/gpl_accepted
>>>> var/ipfire/main/hostname.conf
>>>> var/ipfire/main/hosts
>>>> var/ipfire/main/manualpages
>>>> var/ipfire/main/routing
>>>> var/ipfire/main/security
>>>> var/ipfire/main/send_profile
>>>> var/ipfire/main/settings
>>>> var/ipfire/modem/settings
>>>> var/ipfire/optionsfw/settings
>>>> var/ipfire/ovpn
>>>> var/ipfire/ovpn/ccd.conf
>>>> var/ipfire/ovpn/collectd.vpn
>>>> var/ipfire/ovpn/enable
>>>> var/ipfire/ovpn/server.conf
>>>> var/ipfire/ovpn/settings
>>>> var/ipfire/pakfire/settings
>>>> var/ipfire/ppp
>>>> var/ipfire/ppp/fake-resolv.conf
>>>> var/ipfire/ppp/settings
>>>> var/ipfire/private/cakey.pem
>>>> var/ipfire/proxy
>>>> var/ipfire/proxy/asnbl-helper.conf
>>>> var/ipfire/proxy/cachemgr.conf
>>>> var/ipfire/proxy/enable
>>>> var/ipfire/proxy/settings
>>>> var/ipfire/proxy/squid.conf
>>>> var/ipfire/qos/bin
>>>> var/ipfire/qos/classes
>>>> var/ipfire/qos/level7config
>>>> var/ipfire/qos/portconfig
>>>> var/ipfire/qos/settings
>>>> var/ipfire/qos/subclasses
>>>> var/ipfire/qos/tosconfig
>>>> var/ipfire/remote/enablessh
>>>> var/ipfire/remote/settings
>>>> var/ipfire/sensors/settings
>>>> var/ipfire/suricata/oinkmaster.conf
>>>> var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>> var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>> var/ipfire/suricata/providers-settings
>>>> var/ipfire/suricata/settings
>>>> var/ipfire/suricata/suricata-default-rules.yaml
>>>> var/ipfire/suricata/suricata-dns-servers.yaml
>>>> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>> var/ipfire/suricata/suricata-homenet.yaml
>>>> var/ipfire/suricata/suricata-http-ports.yaml
>>>> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>> var/ipfire/suricata/suricata-used-providers.yaml
>>>> var/ipfire/time/
>>>> var/ipfire/time/counter.conf
>>>> var/ipfire/time/enable
>>>> var/ipfire/time/settime.conf
>>>> var/ipfire/time/settings
>>>> var/ipfire/upnp/settings
>>>> var/ipfire/urlfilter
>>>> var/ipfire/urlfilter/settings
>>>> var/ipfire/urlfilter/squidGuard.conf
>>>> var/ipfire/vpn
>>>> var/ipfire/vpn/config
>>>> var/ipfire/vpn/ipsec.conf
>>>> var/ipfire/vpn/settings
>>>> var/ipfire/wakeonlan/clients.conf
>>>> var/ipfire/wio/wio.conf
>>>> var/ipfire/wireless/config
>>>> var/ipfire/wireless/settings
>>>> var/lib/suricata
>>>> var/log/rrd/collectd
>>>> var/log/rrd/hddshutdown-md127.rrd
>>>> var/log/rrd/hddshutdown-sda.rrd
>>>> var/log/rrd/hddshutdown-sdb.rrd
>>>> var/log/rrd/hddtemp-md127.rrd
>>>> var/log/rrd/hddtemp-sda.rrd
>>>> var/log/rrd/hddtemp-sdb.rrd
>>>> var/log/rrd/wio
>>>> var/log/vnstat
>>>> var/tmp/idsrules-emerging.tar.gz
>>>> var/tmp/idsrules-sslbl_blacklist.rules
>>>> The following are in the previous list but not in this one:-
>>>> /
>>>> /etc/conntrackd/conntrackd.conf
>>>> /etc/ipsec.user-post.conf
>>>> /root/.gitconfig
>>>> /root/.ssh
>>>> Regards,
>>>> Adolf.
>>>> On 29/03/2022 16:01, Adolf Belka wrote:
>>>>> Hi All,
>>>>>
>>>>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>>>>> Hello,
>>>>>>>
>>>>>>> Could you please send the file listing to find out what is being 
>>>>>>> included what shouldn’t?
>>>>> Unfortunately, as I stopped the backup continuing once it had 
>>>>> reached 1.2GB, the file created was not able to be opened. Probably 
>>>>> stopping the backup corrupted it in some way.
>>>>>>>
>>>>>>
>>>>>> Just for easy quick handling, you can use
>>>>>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
>>>>> I gave this a go and here is the output from the command.
>>>>>
>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>> /
>>>>> /etc/conntrackd/conntrackd.conf
>>>>> /etc/group
>>>>> /etc/hosts
>>>>> /etc/hosts.allow
>>>>> /etc/hosts.deny
>>>>> /etc/httpd/server.crt
>>>>> /etc/httpd/server.csr
>>>>> /etc/httpd/server-ecdsa.crt
>>>>> /etc/httpd/server-ecdsa.csr
>>>>> /etc/httpd/server-ecdsa.key
>>>>> /etc/httpd/server.key
>>>>> /etc/ipsec.user.conf
>>>>> /etc/ipsec.user-post.conf
>>>>> /etc/ipsec.user.secrets
>>>>> /etc/logrotate.d
>>>>> /etc/passwd
>>>>> /etc/shadow
>>>>> /etc/squid/squid.conf.local
>>>>> /etc/squid/squid.conf.pre.local
>>>>> /etc/ssh/sshd_config
>>>>> /etc/ssh/ssh_host_ecdsa_key
>>>>> /etc/ssh/ssh_host_ecdsa_key.pub
>>>>> /etc/ssh/ssh_host_ed25519_key
>>>>> /etc/ssh/ssh_host_ed25519_key.pub
>>>>> /etc/ssh/ssh_host_rsa_key
>>>>> /etc/ssh/ssh_host_rsa_key.pub
>>>>> //etc/sudoers
>>>>> /etc/sysconfig/createfiles
>>>>> /etc/sysconfig/firewall.local
>>>>> /etc/sysconfig/lm_sensors
>>>>> /etc/sysconfig/modules
>>>>> /etc/sysconfig/ramdisk
>>>>> /etc/sysconfig/rc
>>>>> /etc/sysconfig/rc.local
>>>>> /etc/unbound
>>>>> //home/ahb
>>>>> /root/.bash_history
>>>>> /root/.gitconfig
>>>>> /root/.ssh
>>>>> /var/ipfire/accounting/settings.conf
>>>>> /var/ipfire/auth/users
>>>>> /var/ipfire/backup/addons/backup
>>>>> /var/ipfire/backup/exclude.user
>>>>> /var/ipfire/backup/include.user
>>>>> /var/ipfire/ca/cacert.pem
>>>>> /var/ipfire/captive/agb.txt
>>>>> /var/ipfire/captive/clients
>>>>> /var/ipfire/captive/coupons
>>>>> /var/ipfire/captive/logo.dat
>>>>> /var/ipfire/captive/settings
>>>>> /var/ipfire/captive/terms.txt
>>>>> /var/ipfire/captive/voucher_out
>>>>> /var/ipfire/certs/hostcert.pem
>>>>> /var/ipfire/certs/hostkey.pem
>>>>> /var/ipfire/certs/phoebevmipseccert.pem
>>>>> /var/ipfire/connscheduler/connscheduler.conf
>>>>> /var/ipfire/crls/cacrl.pem
>>>>> /var/ipfire/cups/cups-browsed.conf
>>>>> /var/ipfire/cups/subscriptions.conf
>>>>> /var/ipfire/ddns/config
>>>>> /var/ipfire/ddns/ddns.conf
>>>>> /var/ipfire/ddns/settings
>>>>> /var/ipfire/dhcp/advoptions
>>>>> /var/ipfire/dhcp/advoptions-list
>>>>> /var/ipfire/dhcpc/dhcpcd.conf
>>>>> /var/ipfire/dhcp/dhcpd.conf
>>>>> /var/ipfire/dhcp/dhcpd.conf.local
>>>>> /var/ipfire/dhcp/enable_blue
>>>>> /var/ipfire/dhcp/enable_green
>>>>> /var/ipfire/dhcp/fixleases
>>>>> /var/ipfire/dhcp/settings
>>>>> /var/ipfire/dma/auth.conf
>>>>> /var/ipfire/dma/dma.conf
>>>>> /var/ipfire/dma/mail.conf
>>>>> /var/ipfire/dns
>>>>> /var/ipfire/dnsforward/config
>>>>> /var/ipfire/dns/settings
>>>>> /var/ipfire/ethernet/aliases
>>>>> /var/ipfire/ethernet/settings
>>>>> /var/ipfire/ethernet/wireless
>>>>> /var/ipfire/extrahd/settings
>>>>> /var/ipfire/firewall
>>>>> /var/ipfire/firewall/config
>>>>> /var/ipfire/firewall/settings
>>>>> /var/ipfire/fwhosts
>>>>> /var/ipfire/isdn/settings
>>>>> /var/ipfire/logging/settings
>>>>> /var/ipfire/mac/settings
>>>>> /var/ipfire/main/firstsetup_ok
>>>>> /var/ipfire/main/gpl_accepted
>>>>> /var/ipfire/main/hostname.conf
>>>>> /var/ipfire/main/hosts
>>>>> /var/ipfire/main/manualpages
>>>>> /var/ipfire/main/routing
>>>>> /var/ipfire/main/security
>>>>> /var/ipfire/main/send_profile
>>>>> /var/ipfire/main/settings
>>>>> /var/ipfire/modem/settings
>>>>> /var/ipfire/optionsfw/settings
>>>>> /var/ipfire/ovpn
>>>>> /var/ipfire/ovpn/ccd.conf
>>>>> /var/ipfire/ovpn/collectd.vpn
>>>>> /var/ipfire/ovpn/enable
>>>>> /var/ipfire/ovpn/server.conf
>>>>> /var/ipfire/ovpn/settings
>>>>> /var/ipfire/pakfire/settings
>>>>> /var/ipfire/ppp
>>>>> /var/ipfire/ppp/fake-resolv.conf
>>>>> /var/ipfire/ppp/settings
>>>>> /var/ipfire/private/cakey.pem
>>>>> /var/ipfire/proxy
>>>>> /var/ipfire/proxy/asnbl-helper.conf
>>>>> /var/ipfire/proxy/cachemgr.conf
>>>>> /var/ipfire/proxy/enable
>>>>> /var/ipfire/proxy/settings
>>>>> /var/ipfire/proxy/squid.conf
>>>>> /var/ipfire/qos/bin
>>>>> /var/ipfire/qos/bin/qos.sh
>>>>> /var/ipfire/qos/classes
>>>>> /var/ipfire/qos/level7config
>>>>> /var/ipfire/qos/portconfig
>>>>> /var/ipfire/qos/settings
>>>>> /var/ipfire/qos/subclasses
>>>>> /var/ipfire/qos/tosconfig
>>>>> /var/ipfire/remote/enablessh
>>>>> /var/ipfire/remote/settings
>>>>> /var/ipfire/sensors/settings
>>>>> /var/ipfire/suricata/oinkmaster.conf
>>>>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>> /var/ipfire/suricata/providers-settings
>>>>> /var/ipfire/suricata/settings
>>>>> /var/ipfire/suricata/suricata-default-rules.yaml
>>>>> /var/ipfire/suricata/suricata-dns-servers.yaml
>>>>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>> /var/ipfire/suricata/suricata-homenet.yaml
>>>>> /var/ipfire/suricata/suricata-http-ports.yaml
>>>>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>> /var/ipfire/suricata/suricata-used-providers.yaml
>>>>> /var/ipfire/time/
>>>>> /var/ipfire/time/counter.conf
>>>>> /var/ipfire/time/enable
>>>>> /var/ipfire/time/settime.conf
>>>>> /var/ipfire/time/settings
>>>>> /var/ipfire/upnp/settings
>>>>> /var/ipfire/urlfilter
>>>>> /var/ipfire/urlfilter/settings
>>>>> /var/ipfire/urlfilter/squidGuard.conf
>>>>> /var/ipfire/vpn
>>>>> /var/ipfire/vpn/config
>>>>> /var/ipfire/vpn/ipsec.conf
>>>>> /var/ipfire/vpn/settings
>>>>> /var/ipfire/wakeonlan/clients.conf
>>>>> /var/ipfire/wio/wio.conf
>>>>> /var/ipfire/wireless/config
>>>>> /var/ipfire/wireless/settings
>>>>> /var/lib/suricata
>>>>> /var/log/rrd/collectd
>>>>> /var/log/rrd/hddshutdown-md127.rrd
>>>>> /var/log/rrd/hddshutdown-sda.rrd
>>>>> /var/log/rrd/hddshutdown-sdb.rrd
>>>>> /var/log/rrd/hddtemp-md127.rrd
>>>>> /var/log/rrd/hddtemp-sda.rrd
>>>>> /var/log/rrd/hddtemp-sdb.rrd
>>>>> /var/log/rrd/wio
>>>>> /var/log/vnstat
>>>>> /var/tmp/idsrules-emerging.tar.gz
>>>>> /var/tmp/idsrules-sslbl_blacklist.rules
>>>>>
>>>>> Regards,
>>>>> Adolf.
>>>>>>
>>>>>> Bernhard
>>>>>>> -Michael
>>>>>>>
>>>>>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka(a)ipfire.org> 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi Michael,
>>>>>>>>
>>>>>>>> Just tried this patch out on my vm testbed system and it still 
>>>>>>>> doesn't work for me. The backup file had got to 1.3GB when I 
>>>>>>>> deleted the backup file as it was still growing. The normal 
>>>>>>>> correct backup file on that vm machine is around 7MB
>>>>>>>>
>>>>>>>> The above was the case for both running it from the WUI or from 
>>>>>>>> the command line from my unprivileged user using sudo backupctrl 
>>>>>>>> exclude
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Adolf.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>>>>> This patch fixes globbing expansion in the backup include file 
>>>>>>>>> list
>>>>>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org>
>>>>>>>>> ---
>>>>>>>>>    config/backup/backup.pl | 8 ++++----
>>>>>>>>>    1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>>>>>
>>>>>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>>>>>>> index a2337cf23..6f9295e94 100644
>>>>>>>>> --- a/config/backup/backup.pl
>>>>>>>>> +++ b/config/backup/backup.pl
>>>>>>>>> @@ -19,6 +19,8 @@
>>>>>>>>> # #
>>>>>>>>> ############################################################################### 
>>>>>>>>>
>>>>>>>>>    +shopt -s nullglob
>>>>>>>>> +
>>>>>>>>>    NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>>>>>      list_addons() {
>>>>>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>>>>>        for include in $@; do
>>>>>>>>>            local file
>>>>>>>>>            while read -r file; do
>>>>>>>>> -            for file in ${file}; do
>>>>>>>>> -                if [ -e "/${file}" ]; then
>>>>>>>>> -                    echo "${file}"
>>>>>>>>> -                fi
>>>>>>>>> +            for file in /${file}; do
>>>>>>>>> +                echo "${file}"
>>>>>>>>>                done
>>>>>>>>>            done < "${include}"
>>>>>>>>>        done | sort -u
>>>>>>>
>>