Re: [PATCH] Core Update 166: Perform spring clean of orphaned files

["Peter Müller" <peter.mueller@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 15631 bytes --]

Hello Michael,

thanks for your reply and the information.

How should we proceed with this patch then?

Thanks, and best regards,
Peter Müller

> Hello,
> 
>> On 21 Mar 2022, at 19:15, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>
>> Hello Michael,
>>
>> thanks for your reply.
>>
>>> Hello,
>>>
>>>> On 21 Mar 2022, at 16:03, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>>
>>>> Hello Michael,
>>>>
>>>> thanks for your reply.
>>>>
>>>>> Hello,
>>>>>
>>>>> This is interesting.
>>>>
>>>> That's a mild way of putting it, I think. :-)
>>>>
>>>>>
>>>>>> On 21 Mar 2022, at 15:35, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>>>>
>>>>>> On an IPFire installation that has been around since Core Update 110
>>>>>> (released April 28, 2017), these files have been identified as being
>>>>>> orphaned, comparing to a fresh installation of Core Update 164.
>>>>>>
>>>>>> To avoid such a list of files agglomerating, persons responsible for a
>>>>>> Core Update should check whether an updated package contains deleted
>>>>>> files in its rootfile. If so, they should be deleted on existing systems
>>>>>> via update.sh as well.
>>>>>>
>>>>>> Signed-off-by: Peter Müller <peter.mueller(a)ipfire.org>
>>>>>> ---
>>>>>> config/rootfiles/core/166/update.sh | 260 +++++++++++++++++++++++++++-
>>>>>> 1 file changed, 258 insertions(+), 2 deletions(-)
>>>>>>
>>>>>> diff --git a/config/rootfiles/core/166/update.sh b/config/rootfiles/core/166/update.sh
>>>>>> index 1370555b0..b556986a5 100644
>>>>>> --- a/config/rootfiles/core/166/update.sh
>>>>>> +++ b/config/rootfiles/core/166/update.sh
>>>>>> @@ -32,13 +32,269 @@ for (( i=1; i<=$core; i++ )); do
>>>>>> done
>>>>>>
>>>>>> # Remove files
>>>>>> -rm -vf \
>>>>>> +rm -rvf \
>>>>>> +	/bin/setserial \
>>>>>> 	/etc/dracut.conf \
>>>>>> +	/etc/fonts/conf.d/30-urw-aliases.conf \
>>>>>> +	/etc/grub.d/README \
>>>>>> +	/etc/rc.d/init.d/networking/red.up/99-geoip-database \
>>>>>> +	/etc/udev/rules.d/99-fuse.rules \
>>>>>> +	/lib/firmware/amd-ucode/microcode_amd.bin.asc \
>>>>>> +	/lib/firmware/amd-ucode/microcode_amd_fam15h.bin.asc \
>>>>>> +	/lib/firmware/amd-ucode/microcode_amd_fam16h.bin.asc \
>>>>>> +	/lib/firmware/amd-ucode/microcode_amd_fam17h.bin.asc \
>>>>>> +	/lib/firmware/ath10k/QCA4019/hw1.0/notice_ath10k_firmware-5.txt \
>>>>>> +	/lib/firmware/ath10k/QCA6174/hw2.1/notice_ath10k_firmware-5.txt \
>>>>>> +	/lib/firmware/ath10k/QCA6174/hw3.0/notice_ath10k_firmware-4.txt \
>>>>>> +	/lib/firmware/ath10k/QCA6174/hw3.0/notice_ath10k_firmware-6.txt \
>>>>>> +	/lib/firmware/ath10k/QCA9377/hw1.0/notice_ath10k_firmware-5.txt \
>>>>>> +	/lib/firmware/ath10k/QCA9377/hw1.0/notice_ath10k_firmware-6.txt \
>>>>>> +	/lib/firmware/ath10k/QCA9887/hw1.0/notice_ath10k_firmware-5.txt \
>>>>>> +	/lib/firmware/ath10k/QCA9888/hw2.0/notice_ath10k_firmware-5.txt \
>>>>>> +	/lib/firmware/ath10k/QCA988X/hw2.0/notice_ath10k_firmware-4.txt \
>>>>>> +	/lib/firmware/ath10k/QCA988X/hw2.0/notice_ath10k_firmware-5.txt \
>>>>>> +	/lib/firmware/ath10k/QCA9984/hw1.0/notice_ath10k_firmware-5.txt \
>>>>>> +	/lib/firmware/ath10k/QCA99X0/hw2.0/notice_ath10k_firmware-5.txt \
>>>>>> +	/lib/firmware/atusb/ChangeLog \
>>>>>> +	/lib/firmware/check_whence.py \
>>>>>> +	/lib/firmware/cis/src \
>>>>>> +	/lib/firmware/copy-firmware.sh \
>>>>>> +	/lib/firmware/cxgb4/t4fw-1.20.8.0.bin \
>>>>>> +	/lib/firmware/cxgb4/t4fw-1.24.3.0.bin \
>>>>>> +	/lib/firmware/cxgb4/t5fw-1.20.8.0.bin \
>>>>>> +	/lib/firmware/cxgb4/t5fw-1.24.3.0.bin \
>>>>>> +	/lib/firmware/cxgb4/t6fw-1.20.8.0.bin \
>>>>>> +	/lib/firmware/cxgb4/t6fw-1.24.3.0.bin \
>>>>>> +	/lib/firmware/GPL-2 \
>>>>>> +	/lib/firmware/GPL-3 \
>>>>>> +	/lib/firmware/isci/README \
>>>>>> +	/lib/firmware/LICENCE.* \
>>>>>> +	/lib/firmware/Makefile \
>>>>>> +	/lib/firmware/qca/NOTICE.txt \
>>>>>> +	/lib/firmware/qcom/NOTICE.txt \
>>>>>> +	/lib/firmware/README \
>>>>>> +	/lib/firmware/WHENCE \
>>>>>> +	/lib/kbd/keymaps/i386/qwerty/fi-latin1.map.gz \
>>>>>> +	/lib/kbd/keymaps/i386/qwerty/fi-latin9.map.gz \
>>>>>> +	/lib/ld-2.29.so \
>>>>>> +	/lib/ld-2.31.so \
>>>>>> +	/lib/ld-2.32.so \
>>>>>
>>>>> These should have been gone before. Ideally by the filesystem cleanup script.
>>>>
>>>> To be honest, I never had a look at this script, and therefore do not have an idea
>>>> what it does and what it does not. I will investigate and get back to you, if necessary.
>>>>
>>>>>> +	/lib/libcap.so \
>>>>>> +	/lib/libhistory.so.5 \
>>>>>> +	/lib/libhistory.so.5.2 \
>>>>>> +	/lib/libip4tc.so.0 \
>>>>>> +	/lib/libip4tc.so.0.1.0 \
>>>>>> +	/lib/libip6tc.so.0 \
>>>>>> +	/lib/libip6tc.so.0.1.0 \
>>>>>> +	/lib/libiptc.so \
>>>>>> +	/lib/libiptc.so.0 \
>>>>>> +	/lib/libiptc.so.0.0.0 \
>>>>>> +	/lib/libnss_nis-2.31.so \
>>>>>> +	/lib/libnss_nisplus-2.31.so \
>>>>>> +	/lib/libnss_nisplus.so.2 \
>>>>>> +	/lib/libnss_nis.so.2 \
>>>>>> +	/lib/libproc-3.2.8.so \
>>>>>> +	/lib/libreadline.so.5 \
>>>>>> +	/lib/libreadline.so.5.2 \
>>>>>> +	/lib/libsysfs.so \
>>>>>> +	/lib/libsysfs.so.1 \
>>>>>> +	/lib/libsysfs.so.1.0.3 \
>>>>>> +	/lib/udev/bluetooth_serial \
>>>>>> +	/lib/udev/rules.d/24-bluetooth.rules \
>>>>>> +	/lib/xtables/libxt_IMQ.so \
>>>>>> 	/opt/pakfire/pakfire-2007.key \
>>>>>> +	/sbin/mount.fuse \
>>>>>> +	/sbin/raw \
>>>>>> +	/sbin/xfs_scrub \
>>>>>> +	/sbin/xfs_scrub_all \
>>>>>> +	/srv/web/ipfire/cgi-bin/bluetooth.cgi \
>>>>>> +	/usr/bin/ez-ipupdate \
>>>>>
>>>>> LOL
>>>>
>>>> ?
>>>>
>>>>>
>>>>>> +	/usr/bin/fusermount \
>>>>>> +	/usr/bin/gawk-5.1.0 \
>>>>>> +	/usr/bin/gcov-dump \
>>>>>> +	/usr/bin/getunimap \
>>>>>> 	/usr/bin/mkinitrd \
>>>>>> +	/usr/bin/mtools \
>>>>>> +	/usr/bin/pango-querymodules \
>>>>>> +	/usr/bin/perl5.30.0 \
>>>>>> +	/usr/bin/setlogcons \
>>>>>> +	/usr/bin/setvesablank \
>>>>>> +	/usr/bin/ulockmgr_server \
>>>>>> +	/usr/include/python2.7 \
>>>>>> +	/usr/lib/cairo \
>>>>>> 	/usr/lib/dracut \
>>>>>> +	/usr/lib/findutils/bigram \
>>>>>> +	/usr/lib/findutils/code \
>>>>>> +	/usr/lib/gawk/testext.so \
>>>>>> +	/usr/lib/itcl4.2.1 \
>>>>>> +	/usr/lib/libasan.so.5 \
>>>>>> +	/usr/lib/libasan.so.5.0.0 \
>>>>>> +	/usr/lib/libbfd-2.32.so \
>>>>>> +	/usr/lib/libbfd-2.34.so \
>>>>>> +	/usr/lib/libbfd-2.35.1.so \
>>>>>> +	/usr/lib/libbind9-9.16.22.so \
>>>>>> +	/usr/lib/libbind9.so.161 \
>>>>>> +	/usr/lib/libbind9.so.161.0.4 \
>>>>>> +	/usr/lib/libblkid.so \
>>>>>> +	/usr/lib/libdns-9.16.22.so \
>>>>>> +	/usr/lib/libdnssec.so.7 \
>>>>>> +	/usr/lib/libdnssec.so.7.0.0 \
>>>>>> +	/usr/lib/libdns.so.* \
>>>>>
>>>>> These should have been caught by the script as well. Any idea why that didn’t happen?
>>>>>
>>>>>> +	/usr/lib/libevent-2.1.so.6 \
>>>>>> +	/usr/lib/libevent-2.1.so.6.0.2 \
>>>>>> +	/usr/lib/libevent_core-2.1.so.6 \
>>>>>> +	/usr/lib/libevent_core-2.1.so.6.0.2 \
>>>>>> +	/usr/lib/libevent_extra-2.1.so.6 \
>>>>>> +	/usr/lib/libevent_extra-2.1.so.6.0.2 \
>>>>>> +	/usr/lib/libevent_openssl-2.1.so.6 \
>>>>>> +	/usr/lib/libevent_openssl-2.1.so.6.0.2 \
>>>>>> +	/usr/lib/libevent_openssl.so \
>>>>>> +	/usr/lib/libevent_pthreads-2.1.so.6 \
>>>>>> +	/usr/lib/libevent_pthreads-2.1.so.6.0.2 \
>>>>>> +	/usr/lib/libevent_pthreads.so \
>>>>>> +	/usr/lib/libexpat.so \
>>>>>> +	/usr/lib/libexslt.so \
>>>>>> +	/usr/lib/libffi.so.6 \
>>>>>> +	/usr/lib/libffi.so.6.0.4 \
>>>>>> +	/usr/lib/libffi.so.7 \
>>>>>> +	/usr/lib/libffi.so.7.1.0 \
>>>>>> +	/usr/lib/libfuse.so \
>>>>>> +	/usr/lib/libfuse.so.2 \
>>>>>> +	/usr/lib/libfuse.so.2.9.7 \
>>>>>> +	/usr/lib/libgdbm_compat.so \
>>>>>> +	/usr/lib/libgdbm_compat.so.3 \
>>>>>> +	/usr/lib/libgdbm_compat.so.3.0.0 \
>>>>>> +	/usr/lib/libgdbm.so \
>>>>>> +	/usr/lib/libgdbm.so.3 \
>>>>>> +	/usr/lib/libgdbm.so.3.0.0 \
>>>>>> +	/usr/lib/libgd.so \
>>>>>> +	/usr/lib/libgd.so.2 \
>>>>>> +	/usr/lib/libgd.so.2.0.0 \
>>>>>> +	/usr/lib/libgettextlib-0.19.8.1.so \
>>>>>> +	/usr/lib/libgettextsrc-0.19.8.1.so \
>>>>>> +	/usr/lib/libhistory.so.6 \
>>>>>> +	/usr/lib/libhistory.so.6.3 \
>>>>>> +	/usr/lib/libhogweed.so.5 \
>>>>>> +	/usr/lib/libhogweed.so.5.0 \
>>>>>> +	/usr/lib/libidn.so \
>>>>>> +	/usr/lib/libidn.so.11 \
>>>>>> +	/usr/lib/libidn.so.11.6.18 \
>>>>>> +	/usr/lib/libirs-9.16.22.so \
>>>>>> +	/usr/lib/libisc-9.16.22.so \
>>>>>> +	/usr/lib/libisccc-9.16.22.so \
>>>>>> +	/usr/lib/libisccc.so.161 \
>>>>>> +	/usr/lib/libisccc.so.161.0.1 \
>>>>>> +	/usr/lib/libisccfg-9.16.22.so \
>>>>>> +	/usr/lib/libisccfg.so.163 \
>>>>>> +	/usr/lib/libisccfg.so.163.0.8 \
>>>>>> +	/usr/lib/libisc.so.1100 \
>>>>>> +	/usr/lib/libisc.so.1100.3.2 \
>>>>>> +	/usr/lib/libisc.so.1104 \
>>>>>> +	/usr/lib/libisc.so.1104.0.0 \
>>>>>> +	/usr/lib/libisc.so.1105 \
>>>>>> +	/usr/lib/libisc.so.1105.1.1 \
>>>>>> +	/usr/lib/libisc.so.1107 \
>>>>>> +	/usr/lib/libisc.so.1107.0.5 \
>>>>>
>>>>> See above.
>>>>>
>>>>>> +	/usr/lib/libixml.so \
>>>>>> +	/usr/lib/libknot.so.10 \
>>>>>> +	/usr/lib/libknot.so.10.0.0 \
>>>>>> +	/usr/lib/libknot.so.11 \
>>>>>> +	/usr/lib/libknot.so.11.0.0 \
>>>>>> +	/usr/lib/libknot.so.9 \
>>>>>> +	/usr/lib/libknot.so.9.0.0 \
>>>>>> +	/usr/lib/liblber-2.3.so.0 \
>>>>>> +	/usr/lib/liblber-2.3.so.0.2.8 \
>>>>>> +	/usr/lib/libldap-2.3.so.0 \
>>>>>> +	/usr/lib/libldap-2.3.so.0.2.8 \
>>>>>> +	/usr/lib/libldap_r-2.3.so.0 \
>>>>>> +	/usr/lib/libldap_r-2.3.so.0.2.8 \
>>>>>> +	/usr/lib/libloc.so.0 \
>>>>>> +	/usr/lib/libloc.so.0.0.0 \
>>>>>> +	/usr/lib/liblua-5.3.so \
>>>>>> +	/usr/lib/liblua.so \
>>>>>> +	/usr/lib/liblwres.so.161 \
>>>>>> +	/usr/lib/liblwres.so.161.0.4 \
>>>>>> +	/usr/lib/libmpfr.so.4 \
>>>>>> +	/usr/lib/libmpfr.so.4.1.5 \
>>>>>> +	/usr/lib/libmpx.so.2 \
>>>>>> +	/usr/lib/libmpx.so.2.0.1 \
>>>>>> +	/usr/lib/libmpxwrappers.so.2 \
>>>>>> +	/usr/lib/libmpxwrappers.so.2.0.1 \
>>>>>> +	/usr/lib/libnettle.so.7 \
>>>>>> +	/usr/lib/libnettle.so.7.0 \
>>>>>> +	/usr/lib/libns-9.16.22.so \
>>>>>> +	/usr/lib/libopcodes-2.32.so \
>>>>>> +	/usr/lib/libopcodes-2.34.so \
>>>>>> +	/usr/lib/libopcodes-2.35.1.so \
>>>>>> +	/usr/lib/libpcre2-posix.so.2 \
>>>>>> +	/usr/lib/libpcre2-posix.so.2.0.3 \
>>>>>> +	/usr/lib/libpng12.so \
>>>>>> +	/usr/lib/libpng12.so.0 \
>>>>>> +	/usr/lib/libpng12.so.0.57.0 \
>>>>>> +	/usr/lib/libpng.so.3 \
>>>>>> +	/usr/lib/libpng.so.3.57.0 \
>>>>>> +	/usr/lib/libpoppler.so.100 \
>>>>>> +	/usr/lib/libpoppler.so.100.0.0 \
>>>>>> +	/usr/lib/libpoppler.so.110 \
>>>>>> +	/usr/lib/libpoppler.so.110.0.0 \
>>>>>> +	/usr/lib/libpoppler.so.111 \
>>>>>> +	/usr/lib/libpoppler.so.111.0.0 \
>>>>>> +	/usr/lib/libpoppler.so.66 \
>>>>>> +	/usr/lib/libpoppler.so.66.0.0 \
>>>>>> +	/usr/lib/libqpdf.so.17 \
>>>>>> +	/usr/lib/libqpdf.so.17.0.0 \
>>>>>> +	/usr/lib/libreadline.so.6 \
>>>>>> +	/usr/lib/libreadline.so.6.3 \
>>>>>> +	/usr/lib/libsensors.so.4 \
>>>>>> +	/usr/lib/libsensors.so.4.4.0 \
>>>>>> +	/usr/lib/libsqlite3.so \
>>>>>> +	/usr/lib/libthreadutil.so \
>>>>>> +	/usr/lib/libthreadutil.so.6 \
>>>>>> +	/usr/lib/libthreadutil.so.6.0.3 \
>>>>>> +	/usr/lib/libulockmgr.so \
>>>>>> +	/usr/lib/libulockmgr.so.1 \
>>>>>> +	/usr/lib/libulockmgr.so.1.0.1 \
>>>>>> +	/usr/lib/libupnp.so \
>>>>>> +	/usr/lib/libuuid.so \
>>>>>> +	/usr/lib/libxml2.so \
>>>>>> +	/usr/lib/libxslt.so \
>>>>>> +	/usr/lib/pango \
>>>>>> +	/usr/lib/perl5/site_perl/5.30.0 \
>>>>>> +	/usr/lib/perl5/site_perl/5.32.1/Net/IP.pm \
>>>>>> +	/usr/lib/python3.8/ensurepip/_bundled/pip-19.2.3-py2.py3-none-any.whl \
>>>>>> +	/usr/lib/python3.8/idlelib/Icons/idle.icns \
>>>>>> +	/usr/lib/python3.8/lib2to3/Grammar3.8.1.final.0.pickle \
>>>>>> +	/usr/lib/python3.8/lib2to3/PatternGrammar3.8.1.final.0.pickle \
>>>>>
>>>>> Why can we not delete /usr/lib/python3.8 as a whole? This should not be needed any more.
>>>>
>>>> Yes, good point. I was running the comparison based on Core Update 164, but with
>>>> Python being updated to 3.10.x in Core Update 165, this folder won't be necessary
>>>> anymore completely.
>>>>
>>>>>
>>>>>> +	/usr/lib/sqlite3.34.0 \
>>>>>> +	/usr/lib/squid/basic_nis_auth \
>>>>>> +	/usr/lib/squid/ext_time_quota_acl \
>>>>>> +	/usr/lib/tcl8/8.4/platform-1.0.14.tm \
>>>>>> +	/usr/lib/tcl8/8.4/platform-1.0.15.tm \
>>>>>> +	/usr/lib/tcl8/8.5/msgcat-1.6.0.tm \
>>>>>> +	/usr/lib/tcl8/8.5/tcltest-2.4.0.tm \
>>>>>> +	/usr/lib/tcl8/8.6/http-2.8.9.tm \
>>>>>> +	/usr/lib/tcl8/8.6/tdbc/sqlite3-1.0.4.tm \
>>>>>> +	/usr/lib/tcl8/8.6/tdbc/sqlite3-1.1.2.tm \
>>>>>
>>>>> Why do we even ship tcl? Is anything reliant on this?
>>>>
>>>> I don't know.
>>>>
>>>>>> +	/usr/lib/tdbc1.1.2 \
>>>>>> +	/usr/lib/tdbcmysql1.1.2 \
>>>>>> +	/usr/lib/tdbcodbc1.1.2 \
>>>>>> +	/usr/lib/tdbcpostgres1.1.2 \
>>>>>> +	/usr/lib/thread2.8.6 \
>>>>>> +	/usr/libexec/xtables-addons \
>>>>>> 	/usr/local/bin/ovpn-ccd-convert \
>>>>>> -	/usr/local/bin/rebuild-initrd
>>>>>> +	/usr/local/bin/rebuild-initrd \
>>>>>> +	/usr/local/bin/xt_geoip_build \
>>>>>> +	/usr/local/bin/xt_geoip_update \
>>>>>> +	/usr/sbin/batctl \
>>>>>> +	/usr/sbin/fbset \
>>>>>> +	/usr/sbin/update-usbids.sh \
>>>>>> +	/usr/sbin/uuidd \
>>>>>> +	/usr/share/doc/fireinfo \
>>>>>> +	/usr/share/GeoIP \
>>>>>> +	/usr/share/zoneinfo/posix/US/Pacific-New \
>>>>>> +	/usr/share/zoneinfo/right/US/Pacific-New \
>>>>>> +	/usr/share/zoneinfo/US/Pacific-New \
>>>>>> +	/var/lib/GeoIP
>>>>>
>>>>> What method did you use to identify all these files?
>>>>
>>>> On the IPFire installation in question, I ran
>>>>
>>>>> find bin/ etc/ lib/ opt/ run/ sbin/ srv/ usr/ var/ | sort > ~/ist
>>>>
>>>> and scp'ed that file onto my workstation. Then, I downloaded the ISO for Core Update 164
>>>> (because I wanted to have something stable, which is why I did not choose C165 for this),
>>>> installed it in a VM, rebooted it, mounted the root partition and executed the same find
>>>> command with a different file name.
>>>
>>> Okay, so this is based on a random system. Not bad, but would be better if we could diff two filelists from two core updates, wouldn’t it?
>>
>> I am not sure if I understood you here: The filelist above contains all leftovers that have
>> agglomerated since Core Update 110. Comparing two filelists would only cover a fraction of
>> it, since this comparison cannot skip any update.
>>
>> However: Do you see a feasible way to obtain such lists, and perhaps scipt the generation of
>> such a list of orphaned files?
> 
> Yes, we could use “git archive” to create a tarball from any branch, so in this case a core update from the past.
> 
> We could then only extract (or directly send to stdout) all files in config/rootfiles/command, sort the output and compare it to what is currently in config/rootfiles/common. Stretch goal is doing this for all architectures :)
> 
> Here is the command for the main file list:
> 
>   git archive origin/core160 | tar --extract --to-stdout config/rootfiles/common | grep -v "^#" | sort -u
> 
> This could be integrated into make.sh :)
> 
> -Michael
> 
>>
>> Thanks, and best regards,
>> Peter Müller
>>
>>>
>>>>
>>>> A
>>>>
>>>>> diff -Naur soll ist | less
>>>>
>>>> (sorry for the German) outputs a long list, which I vetted manually since some add-ons
>>>> were installed on the productive system, and I did not want to pipe the output into the
>>>> rm command blindly.
>>>>
>>>> Thanks, and best regards,
>>>> Peter Müller
>>>>
>>>>>
>>>>> -Michael
>>>>>
>>>>>>
>>>>>> # Delete old 2007 Pakfire key from GPG keyring
>>>>>> export GNUPGHOME="/opt/pakfire/etc/.gnupg"
>>>>>> -- 
>>>>>> 2.34.1
>