Re: [PATCH] backup: Fix broken globbing expansion

[Bernhard Bitsch <bbitsch@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 17994 bytes --]

Just checked the blank line problem.
Pattern is empty , file name list is '/' --> no names produced ( with 
existence check )
Pattern is empty --> produced file name '/' ( without -e check )

Am 29.03.2022 um 20:36 schrieb Bernhard Bitsch:
> Hi all,
> 
> Am 29.03.2022 um 17:23 schrieb Adolf Belka:
>> Hi All,
>>
>> On 29/03/2022 16:22, Michael Tremer wrote:
>>>
>>>> On 29 Mar 2022, at 15:22, Bernhard Bitsch <bbitsch(a)ipfire.org> wrote:
>>>>
>>>> Looks like your include.user file contains absolute paths.
>> That was not the problem but I have corrected that. I would have 
>> expected that when the main include and exclude files were made 
>> relative that the update script would also have checked for any 
>> entries in the include.user and exclude.user files but it looks like 
>> it didn't.th
> According to the source they are chosen.
>>>
>>> Or an empty line maybe?
>> That turned out to be the problem. There was an empty line after the 
>> two lines I had added. You only saw it if you scrolled the pointer 
>> down the file in the editor.
> Interesting. A blank line should produce nothing.
>>
>> Removing that blank line removed the line only with a / and the backup 
>> then successfully ran and created a backup file containing all the 
>> directories expected.
>>
>> The only thing I found is that there was the following tar error message
>>
>> tar: Exiting with failure status due to previous errors
>>
>> The only thing I found in the backup output was several lines such as
>>
>> tar: /etc/squid/squid.conf.local: Cannot stat: No such file or directory
>> tar: /etc/squid/squid.conf.pre.local: Cannot stat: No such file or 
>> directory
>>
>> There were seven lines in total like this and the three I checked were 
>> files that did not exist on my system. I presume that these are files 
>> that should be backed up if present but don't have to be present so 
>> that the error messages are nothing to worry about.
>> There were no error messages when doing the backup via the WUI.
>>
> 
> Good find. ;)
> Michael's patch removes the test of existence, globbing produces 
> existent file names only. But I suppose there are several full names of 
> files, not existing in each implementation.
> Conclusion: Repair the file globbing by the shopt and do the existence 
> check. So all existent files are chosen, that are listed in the include 
> files.
> 
> Regards,
> Bernhard
> 
>> The blank line was only on my vm testbed system but as I clone this 
>> for any testing activities I will have to check all of my existing 
>> vm's to fix this.
>>
>> Glad we resolved this simply. Sorry for the anxiety.
>>
>> Regards,
>>
>> Adolf.
>>
>>>
>>>>
>>>> Am 29.03.2022 um 16:14 schrieb Adolf Belka:
>>>>> Hi All,
>>>>> This is what I get with the pushd/popd version in place on the same 
>>>>> system
>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>> etc/group
>>>>> etc/hosts
>>>>> etc/hosts.allow
>>>>> etc/hosts.deny
>>>>> etc/httpd/server.crt
>>>>> etc/httpd/server.csr
>>>>> etc/httpd/server-ecdsa.crt
>>>>> etc/httpd/server-ecdsa.csr
>>>>> etc/httpd/server-ecdsa.key
>>>>> etc/httpd/server.key
>>>>> etc/ipsec.user.conf
>>>>> etc/ipsec.user.secrets
>>>>> etc/logrotate.d
>>>>> etc/passwd
>>>>> etc/shadow
>>>>> etc/ssh/sshd_config
>>>>> etc/ssh/ssh_host_ecdsa_key
>>>>> etc/ssh/ssh_host_ecdsa_key.pub
>>>>> etc/ssh/ssh_host_ed25519_key
>>>>> etc/ssh/ssh_host_ed25519_key.pub
>>>>> etc/ssh/ssh_host_rsa_key
>>>>> etc/ssh/ssh_host_rsa_key.pub
>>>>> /etc/sudoers
>>>>> etc/sysconfig/createfiles
>>>>> etc/sysconfig/firewall.local
>>>>> etc/sysconfig/lm_sensors
>>>>> etc/sysconfig/modules
>>>>> etc/sysconfig/ramdisk
>>>>> etc/sysconfig/rc
>>>>> etc/sysconfig/rc.local
>>>>> etc/unbound
>>>>> /home/ahb
>>>>> root/.bash_history
>>>>> var/ipfire/accounting/settings.conf
>>>>> var/ipfire/auth/users
>>>>> var/ipfire/backup/addons/backup
>>>>> var/ipfire/backup/exclude.user
>>>>> var/ipfire/backup/include.user
>>>>> var/ipfire/ca/cacert.pem
>>>>> var/ipfire/captive/agb.txt
>>>>> var/ipfire/captive/clients
>>>>> var/ipfire/captive/coupons
>>>>> var/ipfire/captive/logo.dat
>>>>> var/ipfire/captive/settings
>>>>> var/ipfire/captive/terms.txt
>>>>> var/ipfire/captive/voucher_out
>>>>> var/ipfire/certs/hostcert.pem
>>>>> var/ipfire/certs/hostkey.pem
>>>>> var/ipfire/certs/phoebevmipseccert.pem
>>>>> var/ipfire/connscheduler/connscheduler.conf
>>>>> var/ipfire/crls/cacrl.pem
>>>>> var/ipfire/cups/cups-browsed.conf
>>>>> var/ipfire/cups/subscriptions.conf
>>>>> var/ipfire/ddns/config
>>>>> var/ipfire/ddns/ddns.conf
>>>>> var/ipfire/ddns/settings
>>>>> var/ipfire/dhcp/advoptions
>>>>> var/ipfire/dhcp/advoptions-list
>>>>> var/ipfire/dhcpc/dhcpcd.conf
>>>>> var/ipfire/dhcp/dhcpd.conf
>>>>> var/ipfire/dhcp/dhcpd.conf.local
>>>>> var/ipfire/dhcp/enable_blue
>>>>> var/ipfire/dhcp/enable_green
>>>>> var/ipfire/dhcp/fixleases
>>>>> var/ipfire/dhcp/settings
>>>>> var/ipfire/dma/auth.conf
>>>>> var/ipfire/dma/dma.conf
>>>>> var/ipfire/dma/mail.conf
>>>>> var/ipfire/dns
>>>>> var/ipfire/dnsforward/config
>>>>> var/ipfire/dns/settings
>>>>> var/ipfire/ethernet/aliases
>>>>> var/ipfire/ethernet/settings
>>>>> var/ipfire/ethernet/wireless
>>>>> var/ipfire/extrahd/settings
>>>>> var/ipfire/firewall
>>>>> var/ipfire/firewall/config
>>>>> var/ipfire/firewall/settings
>>>>> var/ipfire/fwhosts
>>>>> var/ipfire/isdn/settings
>>>>> var/ipfire/logging/settings
>>>>> var/ipfire/mac/settings
>>>>> var/ipfire/main/firstsetup_ok
>>>>> var/ipfire/main/gpl_accepted
>>>>> var/ipfire/main/hostname.conf
>>>>> var/ipfire/main/hosts
>>>>> var/ipfire/main/manualpages
>>>>> var/ipfire/main/routing
>>>>> var/ipfire/main/security
>>>>> var/ipfire/main/send_profile
>>>>> var/ipfire/main/settings
>>>>> var/ipfire/modem/settings
>>>>> var/ipfire/optionsfw/settings
>>>>> var/ipfire/ovpn
>>>>> var/ipfire/ovpn/ccd.conf
>>>>> var/ipfire/ovpn/collectd.vpn
>>>>> var/ipfire/ovpn/enable
>>>>> var/ipfire/ovpn/server.conf
>>>>> var/ipfire/ovpn/settings
>>>>> var/ipfire/pakfire/settings
>>>>> var/ipfire/ppp
>>>>> var/ipfire/ppp/fake-resolv.conf
>>>>> var/ipfire/ppp/settings
>>>>> var/ipfire/private/cakey.pem
>>>>> var/ipfire/proxy
>>>>> var/ipfire/proxy/asnbl-helper.conf
>>>>> var/ipfire/proxy/cachemgr.conf
>>>>> var/ipfire/proxy/enable
>>>>> var/ipfire/proxy/settings
>>>>> var/ipfire/proxy/squid.conf
>>>>> var/ipfire/qos/bin
>>>>> var/ipfire/qos/classes
>>>>> var/ipfire/qos/level7config
>>>>> var/ipfire/qos/portconfig
>>>>> var/ipfire/qos/settings
>>>>> var/ipfire/qos/subclasses
>>>>> var/ipfire/qos/tosconfig
>>>>> var/ipfire/remote/enablessh
>>>>> var/ipfire/remote/settings
>>>>> var/ipfire/sensors/settings
>>>>> var/ipfire/suricata/oinkmaster.conf
>>>>> var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>> var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>> var/ipfire/suricata/providers-settings
>>>>> var/ipfire/suricata/settings
>>>>> var/ipfire/suricata/suricata-default-rules.yaml
>>>>> var/ipfire/suricata/suricata-dns-servers.yaml
>>>>> var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>> var/ipfire/suricata/suricata-homenet.yaml
>>>>> var/ipfire/suricata/suricata-http-ports.yaml
>>>>> var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>> var/ipfire/suricata/suricata-used-providers.yaml
>>>>> var/ipfire/time/
>>>>> var/ipfire/time/counter.conf
>>>>> var/ipfire/time/enable
>>>>> var/ipfire/time/settime.conf
>>>>> var/ipfire/time/settings
>>>>> var/ipfire/upnp/settings
>>>>> var/ipfire/urlfilter
>>>>> var/ipfire/urlfilter/settings
>>>>> var/ipfire/urlfilter/squidGuard.conf
>>>>> var/ipfire/vpn
>>>>> var/ipfire/vpn/config
>>>>> var/ipfire/vpn/ipsec.conf
>>>>> var/ipfire/vpn/settings
>>>>> var/ipfire/wakeonlan/clients.conf
>>>>> var/ipfire/wio/wio.conf
>>>>> var/ipfire/wireless/config
>>>>> var/ipfire/wireless/settings
>>>>> var/lib/suricata
>>>>> var/log/rrd/collectd
>>>>> var/log/rrd/hddshutdown-md127.rrd
>>>>> var/log/rrd/hddshutdown-sda.rrd
>>>>> var/log/rrd/hddshutdown-sdb.rrd
>>>>> var/log/rrd/hddtemp-md127.rrd
>>>>> var/log/rrd/hddtemp-sda.rrd
>>>>> var/log/rrd/hddtemp-sdb.rrd
>>>>> var/log/rrd/wio
>>>>> var/log/vnstat
>>>>> var/tmp/idsrules-emerging.tar.gz
>>>>> var/tmp/idsrules-sslbl_blacklist.rules
>>>>> The following are in the previous list but not in this one:-
>>>>> /
>>>>> /etc/conntrackd/conntrackd.conf
>>>>> /etc/ipsec.user-post.conf
>>>>> /root/.gitconfig
>>>>> /root/.ssh
>>>>> Regards,
>>>>> Adolf.
>>>>> On 29/03/2022 16:01, Adolf Belka wrote:
>>>>>> Hi All,
>>>>>>
>>>>>> On 29/03/2022 15:36, Bernhard Bitsch wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Am 29.03.2022 um 15:11 schrieb Michael Tremer:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> Could you please send the file listing to find out what is being 
>>>>>>>> included what shouldn’t?
>>>>>> Unfortunately, as I stopped the backup continuing once it had 
>>>>>> reached 1.2GB, the file created was not able to be opened. 
>>>>>> Probably stopping the backup corrupted it in some way.
>>>>>>>>
>>>>>>>
>>>>>>> Just for easy quick handling, you can use
>>>>>>> '/var/ipfire/backup/bin/backup.pl list' to show the files included.
>>>>>> I gave this a go and here is the output from the command.
>>>>>>
>>>>>> -bash-5.1$ sudo /var/ipfire/backup/bin/backup.pl list
>>>>>> /
>>>>>> /etc/conntrackd/conntrackd.conf
>>>>>> /etc/group
>>>>>> /etc/hosts
>>>>>> /etc/hosts.allow
>>>>>> /etc/hosts.deny
>>>>>> /etc/httpd/server.crt
>>>>>> /etc/httpd/server.csr
>>>>>> /etc/httpd/server-ecdsa.crt
>>>>>> /etc/httpd/server-ecdsa.csr
>>>>>> /etc/httpd/server-ecdsa.key
>>>>>> /etc/httpd/server.key
>>>>>> /etc/ipsec.user.conf
>>>>>> /etc/ipsec.user-post.conf
>>>>>> /etc/ipsec.user.secrets
>>>>>> /etc/logrotate.d
>>>>>> /etc/passwd
>>>>>> /etc/shadow
>>>>>> /etc/squid/squid.conf.local
>>>>>> /etc/squid/squid.conf.pre.local
>>>>>> /etc/ssh/sshd_config
>>>>>> /etc/ssh/ssh_host_ecdsa_key
>>>>>> /etc/ssh/ssh_host_ecdsa_key.pub
>>>>>> /etc/ssh/ssh_host_ed25519_key
>>>>>> /etc/ssh/ssh_host_ed25519_key.pub
>>>>>> /etc/ssh/ssh_host_rsa_key
>>>>>> /etc/ssh/ssh_host_rsa_key.pub
>>>>>> //etc/sudoers
>>>>>> /etc/sysconfig/createfiles
>>>>>> /etc/sysconfig/firewall.local
>>>>>> /etc/sysconfig/lm_sensors
>>>>>> /etc/sysconfig/modules
>>>>>> /etc/sysconfig/ramdisk
>>>>>> /etc/sysconfig/rc
>>>>>> /etc/sysconfig/rc.local
>>>>>> /etc/unbound
>>>>>> //home/ahb
>>>>>> /root/.bash_history
>>>>>> /root/.gitconfig
>>>>>> /root/.ssh
>>>>>> /var/ipfire/accounting/settings.conf
>>>>>> /var/ipfire/auth/users
>>>>>> /var/ipfire/backup/addons/backup
>>>>>> /var/ipfire/backup/exclude.user
>>>>>> /var/ipfire/backup/include.user
>>>>>> /var/ipfire/ca/cacert.pem
>>>>>> /var/ipfire/captive/agb.txt
>>>>>> /var/ipfire/captive/clients
>>>>>> /var/ipfire/captive/coupons
>>>>>> /var/ipfire/captive/logo.dat
>>>>>> /var/ipfire/captive/settings
>>>>>> /var/ipfire/captive/terms.txt
>>>>>> /var/ipfire/captive/voucher_out
>>>>>> /var/ipfire/certs/hostcert.pem
>>>>>> /var/ipfire/certs/hostkey.pem
>>>>>> /var/ipfire/certs/phoebevmipseccert.pem
>>>>>> /var/ipfire/connscheduler/connscheduler.conf
>>>>>> /var/ipfire/crls/cacrl.pem
>>>>>> /var/ipfire/cups/cups-browsed.conf
>>>>>> /var/ipfire/cups/subscriptions.conf
>>>>>> /var/ipfire/ddns/config
>>>>>> /var/ipfire/ddns/ddns.conf
>>>>>> /var/ipfire/ddns/settings
>>>>>> /var/ipfire/dhcp/advoptions
>>>>>> /var/ipfire/dhcp/advoptions-list
>>>>>> /var/ipfire/dhcpc/dhcpcd.conf
>>>>>> /var/ipfire/dhcp/dhcpd.conf
>>>>>> /var/ipfire/dhcp/dhcpd.conf.local
>>>>>> /var/ipfire/dhcp/enable_blue
>>>>>> /var/ipfire/dhcp/enable_green
>>>>>> /var/ipfire/dhcp/fixleases
>>>>>> /var/ipfire/dhcp/settings
>>>>>> /var/ipfire/dma/auth.conf
>>>>>> /var/ipfire/dma/dma.conf
>>>>>> /var/ipfire/dma/mail.conf
>>>>>> /var/ipfire/dns
>>>>>> /var/ipfire/dnsforward/config
>>>>>> /var/ipfire/dns/settings
>>>>>> /var/ipfire/ethernet/aliases
>>>>>> /var/ipfire/ethernet/settings
>>>>>> /var/ipfire/ethernet/wireless
>>>>>> /var/ipfire/extrahd/settings
>>>>>> /var/ipfire/firewall
>>>>>> /var/ipfire/firewall/config
>>>>>> /var/ipfire/firewall/settings
>>>>>> /var/ipfire/fwhosts
>>>>>> /var/ipfire/isdn/settings
>>>>>> /var/ipfire/logging/settings
>>>>>> /var/ipfire/mac/settings
>>>>>> /var/ipfire/main/firstsetup_ok
>>>>>> /var/ipfire/main/gpl_accepted
>>>>>> /var/ipfire/main/hostname.conf
>>>>>> /var/ipfire/main/hosts
>>>>>> /var/ipfire/main/manualpages
>>>>>> /var/ipfire/main/routing
>>>>>> /var/ipfire/main/security
>>>>>> /var/ipfire/main/send_profile
>>>>>> /var/ipfire/main/settings
>>>>>> /var/ipfire/modem/settings
>>>>>> /var/ipfire/optionsfw/settings
>>>>>> /var/ipfire/ovpn
>>>>>> /var/ipfire/ovpn/ccd.conf
>>>>>> /var/ipfire/ovpn/collectd.vpn
>>>>>> /var/ipfire/ovpn/enable
>>>>>> /var/ipfire/ovpn/server.conf
>>>>>> /var/ipfire/ovpn/settings
>>>>>> /var/ipfire/pakfire/settings
>>>>>> /var/ipfire/ppp
>>>>>> /var/ipfire/ppp/fake-resolv.conf
>>>>>> /var/ipfire/ppp/settings
>>>>>> /var/ipfire/private/cakey.pem
>>>>>> /var/ipfire/proxy
>>>>>> /var/ipfire/proxy/asnbl-helper.conf
>>>>>> /var/ipfire/proxy/cachemgr.conf
>>>>>> /var/ipfire/proxy/enable
>>>>>> /var/ipfire/proxy/settings
>>>>>> /var/ipfire/proxy/squid.conf
>>>>>> /var/ipfire/qos/bin
>>>>>> /var/ipfire/qos/bin/qos.sh
>>>>>> /var/ipfire/qos/classes
>>>>>> /var/ipfire/qos/level7config
>>>>>> /var/ipfire/qos/portconfig
>>>>>> /var/ipfire/qos/settings
>>>>>> /var/ipfire/qos/subclasses
>>>>>> /var/ipfire/qos/tosconfig
>>>>>> /var/ipfire/remote/enablessh
>>>>>> /var/ipfire/remote/settings
>>>>>> /var/ipfire/sensors/settings
>>>>>> /var/ipfire/suricata/oinkmaster.conf
>>>>>> /var/ipfire/suricata/oinkmaster-modify-sids.conf
>>>>>> /var/ipfire/suricata/oinkmaster-provider-includes.conf
>>>>>> /var/ipfire/suricata/providers-settings
>>>>>> /var/ipfire/suricata/settings
>>>>>> /var/ipfire/suricata/suricata-default-rules.yaml
>>>>>> /var/ipfire/suricata/suricata-dns-servers.yaml
>>>>>> /var/ipfire/suricata/suricata-emerging-used-rulefiles.yaml
>>>>>> /var/ipfire/suricata/suricata-homenet.yaml
>>>>>> /var/ipfire/suricata/suricata-http-ports.yaml
>>>>>> /var/ipfire/suricata/suricata-sslbl_blacklist-used-rulefiles.yaml
>>>>>> /var/ipfire/suricata/suricata-used-providers.yaml
>>>>>> /var/ipfire/time/
>>>>>> /var/ipfire/time/counter.conf
>>>>>> /var/ipfire/time/enable
>>>>>> /var/ipfire/time/settime.conf
>>>>>> /var/ipfire/time/settings
>>>>>> /var/ipfire/upnp/settings
>>>>>> /var/ipfire/urlfilter
>>>>>> /var/ipfire/urlfilter/settings
>>>>>> /var/ipfire/urlfilter/squidGuard.conf
>>>>>> /var/ipfire/vpn
>>>>>> /var/ipfire/vpn/config
>>>>>> /var/ipfire/vpn/ipsec.conf
>>>>>> /var/ipfire/vpn/settings
>>>>>> /var/ipfire/wakeonlan/clients.conf
>>>>>> /var/ipfire/wio/wio.conf
>>>>>> /var/ipfire/wireless/config
>>>>>> /var/ipfire/wireless/settings
>>>>>> /var/lib/suricata
>>>>>> /var/log/rrd/collectd
>>>>>> /var/log/rrd/hddshutdown-md127.rrd
>>>>>> /var/log/rrd/hddshutdown-sda.rrd
>>>>>> /var/log/rrd/hddshutdown-sdb.rrd
>>>>>> /var/log/rrd/hddtemp-md127.rrd
>>>>>> /var/log/rrd/hddtemp-sda.rrd
>>>>>> /var/log/rrd/hddtemp-sdb.rrd
>>>>>> /var/log/rrd/wio
>>>>>> /var/log/vnstat
>>>>>> /var/tmp/idsrules-emerging.tar.gz
>>>>>> /var/tmp/idsrules-sslbl_blacklist.rules
>>>>>>
>>>>>> Regards,
>>>>>> Adolf.
>>>>>>>
>>>>>>> Bernhard
>>>>>>>> -Michael
>>>>>>>>
>>>>>>>>> On 29 Mar 2022, at 14:10, Adolf Belka <adolf.belka(a)ipfire.org> 
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi Michael,
>>>>>>>>>
>>>>>>>>> Just tried this patch out on my vm testbed system and it still 
>>>>>>>>> doesn't work for me. The backup file had got to 1.3GB when I 
>>>>>>>>> deleted the backup file as it was still growing. The normal 
>>>>>>>>> correct backup file on that vm machine is around 7MB
>>>>>>>>>
>>>>>>>>> The above was the case for both running it from the WUI or from 
>>>>>>>>> the command line from my unprivileged user using sudo 
>>>>>>>>> backupctrl exclude
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Adolf.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 29/03/2022 14:27, Michael Tremer wrote:
>>>>>>>>>> This patch fixes globbing expansion in the backup include file 
>>>>>>>>>> list
>>>>>>>>>> which got broken in c7e0d73e7cfd7be95db9d0a5f3392b8241813d5b.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org>
>>>>>>>>>> ---
>>>>>>>>>>    config/backup/backup.pl | 8 ++++----
>>>>>>>>>>    1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>>>>>>
>>>>>>>>>> diff --git a/config/backup/backup.pl b/config/backup/backup.pl
>>>>>>>>>> index a2337cf23..6f9295e94 100644
>>>>>>>>>> --- a/config/backup/backup.pl
>>>>>>>>>> +++ b/config/backup/backup.pl
>>>>>>>>>> @@ -19,6 +19,8 @@
>>>>>>>>>> # #
>>>>>>>>>> ############################################################################### 
>>>>>>>>>>
>>>>>>>>>>    +shopt -s nullglob
>>>>>>>>>> +
>>>>>>>>>>    NOW="$(date "+%Y-%m-%d-%H:%M")"
>>>>>>>>>>      list_addons() {
>>>>>>>>>> @@ -38,10 +40,8 @@ process_includes() {
>>>>>>>>>>        for include in $@; do
>>>>>>>>>>            local file
>>>>>>>>>>            while read -r file; do
>>>>>>>>>> -            for file in ${file}; do
>>>>>>>>>> -                if [ -e "/${file}" ]; then
>>>>>>>>>> -                    echo "${file}"
>>>>>>>>>> -                fi
>>>>>>>>>> +            for file in /${file}; do
>>>>>>>>>> +                echo "${file}"
>>>>>>>>>>                done
>>>>>>>>>>            done < "${include}"
>>>>>>>>>>        done | sort -u
>>>>>>>>
>>>