From mboxrd@z Thu Jan 1 00:00:00 1970 From: Peter =?utf-8?q?M=C3=BCller?= To: development@lists.ipfire.org Subject: Re: [PATCH] Core Update 166: Perform spring clean of orphaned files Date: Tue, 22 Mar 2022 14:59:27 +0000 Message-ID: <2e62d533-1c64-2efc-d6bf-2efc2b839f72@ipfire.org> In-Reply-To: <83EBB518-E64D-41D8-9EBB-A498FDB3096D@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============3784102029282667662==" List-Id: --===============3784102029282667662== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello Michael, thanks for your reply and the information. How should we proceed with this patch then? Thanks, and best regards, Peter M=C3=BCller > Hello, >=20 >> On 21 Mar 2022, at 19:15, Peter M=C3=BCller w= rote: >> >> Hello Michael, >> >> thanks for your reply. >> >>> Hello, >>> >>>> On 21 Mar 2022, at 16:03, Peter M=C3=BCller = wrote: >>>> >>>> Hello Michael, >>>> >>>> thanks for your reply. >>>> >>>>> Hello, >>>>> >>>>> This is interesting. >>>> >>>> That's a mild way of putting it, I think. :-) >>>> >>>>> >>>>>> On 21 Mar 2022, at 15:35, Peter M=C3=BCller wrote: >>>>>> >>>>>> On an IPFire installation that has been around since Core Update 110 >>>>>> (released April 28, 2017), these files have been identified as being >>>>>> orphaned, comparing to a fresh installation of Core Update 164. >>>>>> >>>>>> To avoid such a list of files agglomerating, persons responsible for a >>>>>> Core Update should check whether an updated package contains deleted >>>>>> files in its rootfile. If so, they should be deleted on existing syste= ms >>>>>> via update.sh as well. >>>>>> >>>>>> Signed-off-by: Peter M=C3=BCller >>>>>> --- >>>>>> config/rootfiles/core/166/update.sh | 260 +++++++++++++++++++++++++++- >>>>>> 1 file changed, 258 insertions(+), 2 deletions(-) >>>>>> >>>>>> diff --git a/config/rootfiles/core/166/update.sh b/config/rootfiles/co= re/166/update.sh >>>>>> index 1370555b0..b556986a5 100644 >>>>>> --- a/config/rootfiles/core/166/update.sh >>>>>> +++ b/config/rootfiles/core/166/update.sh >>>>>> @@ -32,13 +32,269 @@ for (( i=3D1; i<=3D$core; i++ )); do >>>>>> done >>>>>> >>>>>> # Remove files >>>>>> -rm -vf \ >>>>>> +rm -rvf \ >>>>>> + /bin/setserial \ >>>>>> /etc/dracut.conf \ >>>>>> + /etc/fonts/conf.d/30-urw-aliases.conf \ >>>>>> + /etc/grub.d/README \ >>>>>> + /etc/rc.d/init.d/networking/red.up/99-geoip-database \ >>>>>> + /etc/udev/rules.d/99-fuse.rules \ >>>>>> + /lib/firmware/amd-ucode/microcode_amd.bin.asc \ >>>>>> + /lib/firmware/amd-ucode/microcode_amd_fam15h.bin.asc \ >>>>>> + /lib/firmware/amd-ucode/microcode_amd_fam16h.bin.asc \ >>>>>> + /lib/firmware/amd-ucode/microcode_amd_fam17h.bin.asc \ >>>>>> + /lib/firmware/ath10k/QCA4019/hw1.0/notice_ath10k_firmware-5.txt \ >>>>>> + /lib/firmware/ath10k/QCA6174/hw2.1/notice_ath10k_firmware-5.txt \ >>>>>> + /lib/firmware/ath10k/QCA6174/hw3.0/notice_ath10k_firmware-4.txt \ >>>>>> + /lib/firmware/ath10k/QCA6174/hw3.0/notice_ath10k_firmware-6.txt \ >>>>>> + /lib/firmware/ath10k/QCA9377/hw1.0/notice_ath10k_firmware-5.txt \ >>>>>> + /lib/firmware/ath10k/QCA9377/hw1.0/notice_ath10k_firmware-6.txt \ >>>>>> + /lib/firmware/ath10k/QCA9887/hw1.0/notice_ath10k_firmware-5.txt \ >>>>>> + /lib/firmware/ath10k/QCA9888/hw2.0/notice_ath10k_firmware-5.txt \ >>>>>> + /lib/firmware/ath10k/QCA988X/hw2.0/notice_ath10k_firmware-4.txt \ >>>>>> + /lib/firmware/ath10k/QCA988X/hw2.0/notice_ath10k_firmware-5.txt \ >>>>>> + /lib/firmware/ath10k/QCA9984/hw1.0/notice_ath10k_firmware-5.txt \ >>>>>> + /lib/firmware/ath10k/QCA99X0/hw2.0/notice_ath10k_firmware-5.txt \ >>>>>> + /lib/firmware/atusb/ChangeLog \ >>>>>> + /lib/firmware/check_whence.py \ >>>>>> + /lib/firmware/cis/src \ >>>>>> + /lib/firmware/copy-firmware.sh \ >>>>>> + /lib/firmware/cxgb4/t4fw-1.20.8.0.bin \ >>>>>> + /lib/firmware/cxgb4/t4fw-1.24.3.0.bin \ >>>>>> + /lib/firmware/cxgb4/t5fw-1.20.8.0.bin \ >>>>>> + /lib/firmware/cxgb4/t5fw-1.24.3.0.bin \ >>>>>> + /lib/firmware/cxgb4/t6fw-1.20.8.0.bin \ >>>>>> + /lib/firmware/cxgb4/t6fw-1.24.3.0.bin \ >>>>>> + /lib/firmware/GPL-2 \ >>>>>> + /lib/firmware/GPL-3 \ >>>>>> + /lib/firmware/isci/README \ >>>>>> + /lib/firmware/LICENCE.* \ >>>>>> + /lib/firmware/Makefile \ >>>>>> + /lib/firmware/qca/NOTICE.txt \ >>>>>> + /lib/firmware/qcom/NOTICE.txt \ >>>>>> + /lib/firmware/README \ >>>>>> + /lib/firmware/WHENCE \ >>>>>> + /lib/kbd/keymaps/i386/qwerty/fi-latin1.map.gz \ >>>>>> + /lib/kbd/keymaps/i386/qwerty/fi-latin9.map.gz \ >>>>>> + /lib/ld-2.29.so \ >>>>>> + /lib/ld-2.31.so \ >>>>>> + /lib/ld-2.32.so \ >>>>> >>>>> These should have been gone before. Ideally by the filesystem cleanup s= cript. >>>> >>>> To be honest, I never had a look at this script, and therefore do not ha= ve an idea >>>> what it does and what it does not. I will investigate and get back to yo= u, if necessary. >>>> >>>>>> + /lib/libcap.so \ >>>>>> + /lib/libhistory.so.5 \ >>>>>> + /lib/libhistory.so.5.2 \ >>>>>> + /lib/libip4tc.so.0 \ >>>>>> + /lib/libip4tc.so.0.1.0 \ >>>>>> + /lib/libip6tc.so.0 \ >>>>>> + /lib/libip6tc.so.0.1.0 \ >>>>>> + /lib/libiptc.so \ >>>>>> + /lib/libiptc.so.0 \ >>>>>> + /lib/libiptc.so.0.0.0 \ >>>>>> + /lib/libnss_nis-2.31.so \ >>>>>> + /lib/libnss_nisplus-2.31.so \ >>>>>> + /lib/libnss_nisplus.so.2 \ >>>>>> + /lib/libnss_nis.so.2 \ >>>>>> + /lib/libproc-3.2.8.so \ >>>>>> + /lib/libreadline.so.5 \ >>>>>> + /lib/libreadline.so.5.2 \ >>>>>> + /lib/libsysfs.so \ >>>>>> + /lib/libsysfs.so.1 \ >>>>>> + /lib/libsysfs.so.1.0.3 \ >>>>>> + /lib/udev/bluetooth_serial \ >>>>>> + /lib/udev/rules.d/24-bluetooth.rules \ >>>>>> + /lib/xtables/libxt_IMQ.so \ >>>>>> /opt/pakfire/pakfire-2007.key \ >>>>>> + /sbin/mount.fuse \ >>>>>> + /sbin/raw \ >>>>>> + /sbin/xfs_scrub \ >>>>>> + /sbin/xfs_scrub_all \ >>>>>> + /srv/web/ipfire/cgi-bin/bluetooth.cgi \ >>>>>> + /usr/bin/ez-ipupdate \ >>>>> >>>>> LOL >>>> >>>> ? >>>> >>>>> >>>>>> + /usr/bin/fusermount \ >>>>>> + /usr/bin/gawk-5.1.0 \ >>>>>> + /usr/bin/gcov-dump \ >>>>>> + /usr/bin/getunimap \ >>>>>> /usr/bin/mkinitrd \ >>>>>> + /usr/bin/mtools \ >>>>>> + /usr/bin/pango-querymodules \ >>>>>> + /usr/bin/perl5.30.0 \ >>>>>> + /usr/bin/setlogcons \ >>>>>> + /usr/bin/setvesablank \ >>>>>> + /usr/bin/ulockmgr_server \ >>>>>> + /usr/include/python2.7 \ >>>>>> + /usr/lib/cairo \ >>>>>> /usr/lib/dracut \ >>>>>> + /usr/lib/findutils/bigram \ >>>>>> + /usr/lib/findutils/code \ >>>>>> + /usr/lib/gawk/testext.so \ >>>>>> + /usr/lib/itcl4.2.1 \ >>>>>> + /usr/lib/libasan.so.5 \ >>>>>> + /usr/lib/libasan.so.5.0.0 \ >>>>>> + /usr/lib/libbfd-2.32.so \ >>>>>> + /usr/lib/libbfd-2.34.so \ >>>>>> + /usr/lib/libbfd-2.35.1.so \ >>>>>> + /usr/lib/libbind9-9.16.22.so \ >>>>>> + /usr/lib/libbind9.so.161 \ >>>>>> + /usr/lib/libbind9.so.161.0.4 \ >>>>>> + /usr/lib/libblkid.so \ >>>>>> + /usr/lib/libdns-9.16.22.so \ >>>>>> + /usr/lib/libdnssec.so.7 \ >>>>>> + /usr/lib/libdnssec.so.7.0.0 \ >>>>>> + /usr/lib/libdns.so.* \ >>>>> >>>>> These should have been caught by the script as well. Any idea why that = didn=E2=80=99t happen? >>>>> >>>>>> + /usr/lib/libevent-2.1.so.6 \ >>>>>> + /usr/lib/libevent-2.1.so.6.0.2 \ >>>>>> + /usr/lib/libevent_core-2.1.so.6 \ >>>>>> + /usr/lib/libevent_core-2.1.so.6.0.2 \ >>>>>> + /usr/lib/libevent_extra-2.1.so.6 \ >>>>>> + /usr/lib/libevent_extra-2.1.so.6.0.2 \ >>>>>> + /usr/lib/libevent_openssl-2.1.so.6 \ >>>>>> + /usr/lib/libevent_openssl-2.1.so.6.0.2 \ >>>>>> + /usr/lib/libevent_openssl.so \ >>>>>> + /usr/lib/libevent_pthreads-2.1.so.6 \ >>>>>> + /usr/lib/libevent_pthreads-2.1.so.6.0.2 \ >>>>>> + /usr/lib/libevent_pthreads.so \ >>>>>> + /usr/lib/libexpat.so \ >>>>>> + /usr/lib/libexslt.so \ >>>>>> + /usr/lib/libffi.so.6 \ >>>>>> + /usr/lib/libffi.so.6.0.4 \ >>>>>> + /usr/lib/libffi.so.7 \ >>>>>> + /usr/lib/libffi.so.7.1.0 \ >>>>>> + /usr/lib/libfuse.so \ >>>>>> + /usr/lib/libfuse.so.2 \ >>>>>> + /usr/lib/libfuse.so.2.9.7 \ >>>>>> + /usr/lib/libgdbm_compat.so \ >>>>>> + /usr/lib/libgdbm_compat.so.3 \ >>>>>> + /usr/lib/libgdbm_compat.so.3.0.0 \ >>>>>> + /usr/lib/libgdbm.so \ >>>>>> + /usr/lib/libgdbm.so.3 \ >>>>>> + /usr/lib/libgdbm.so.3.0.0 \ >>>>>> + /usr/lib/libgd.so \ >>>>>> + /usr/lib/libgd.so.2 \ >>>>>> + /usr/lib/libgd.so.2.0.0 \ >>>>>> + /usr/lib/libgettextlib-0.19.8.1.so \ >>>>>> + /usr/lib/libgettextsrc-0.19.8.1.so \ >>>>>> + /usr/lib/libhistory.so.6 \ >>>>>> + /usr/lib/libhistory.so.6.3 \ >>>>>> + /usr/lib/libhogweed.so.5 \ >>>>>> + /usr/lib/libhogweed.so.5.0 \ >>>>>> + /usr/lib/libidn.so \ >>>>>> + /usr/lib/libidn.so.11 \ >>>>>> + /usr/lib/libidn.so.11.6.18 \ >>>>>> + /usr/lib/libirs-9.16.22.so \ >>>>>> + /usr/lib/libisc-9.16.22.so \ >>>>>> + /usr/lib/libisccc-9.16.22.so \ >>>>>> + /usr/lib/libisccc.so.161 \ >>>>>> + /usr/lib/libisccc.so.161.0.1 \ >>>>>> + /usr/lib/libisccfg-9.16.22.so \ >>>>>> + /usr/lib/libisccfg.so.163 \ >>>>>> + /usr/lib/libisccfg.so.163.0.8 \ >>>>>> + /usr/lib/libisc.so.1100 \ >>>>>> + /usr/lib/libisc.so.1100.3.2 \ >>>>>> + /usr/lib/libisc.so.1104 \ >>>>>> + /usr/lib/libisc.so.1104.0.0 \ >>>>>> + /usr/lib/libisc.so.1105 \ >>>>>> + /usr/lib/libisc.so.1105.1.1 \ >>>>>> + /usr/lib/libisc.so.1107 \ >>>>>> + /usr/lib/libisc.so.1107.0.5 \ >>>>> >>>>> See above. >>>>> >>>>>> + /usr/lib/libixml.so \ >>>>>> + /usr/lib/libknot.so.10 \ >>>>>> + /usr/lib/libknot.so.10.0.0 \ >>>>>> + /usr/lib/libknot.so.11 \ >>>>>> + /usr/lib/libknot.so.11.0.0 \ >>>>>> + /usr/lib/libknot.so.9 \ >>>>>> + /usr/lib/libknot.so.9.0.0 \ >>>>>> + /usr/lib/liblber-2.3.so.0 \ >>>>>> + /usr/lib/liblber-2.3.so.0.2.8 \ >>>>>> + /usr/lib/libldap-2.3.so.0 \ >>>>>> + /usr/lib/libldap-2.3.so.0.2.8 \ >>>>>> + /usr/lib/libldap_r-2.3.so.0 \ >>>>>> + /usr/lib/libldap_r-2.3.so.0.2.8 \ >>>>>> + /usr/lib/libloc.so.0 \ >>>>>> + /usr/lib/libloc.so.0.0.0 \ >>>>>> + /usr/lib/liblua-5.3.so \ >>>>>> + /usr/lib/liblua.so \ >>>>>> + /usr/lib/liblwres.so.161 \ >>>>>> + /usr/lib/liblwres.so.161.0.4 \ >>>>>> + /usr/lib/libmpfr.so.4 \ >>>>>> + /usr/lib/libmpfr.so.4.1.5 \ >>>>>> + /usr/lib/libmpx.so.2 \ >>>>>> + /usr/lib/libmpx.so.2.0.1 \ >>>>>> + /usr/lib/libmpxwrappers.so.2 \ >>>>>> + /usr/lib/libmpxwrappers.so.2.0.1 \ >>>>>> + /usr/lib/libnettle.so.7 \ >>>>>> + /usr/lib/libnettle.so.7.0 \ >>>>>> + /usr/lib/libns-9.16.22.so \ >>>>>> + /usr/lib/libopcodes-2.32.so \ >>>>>> + /usr/lib/libopcodes-2.34.so \ >>>>>> + /usr/lib/libopcodes-2.35.1.so \ >>>>>> + /usr/lib/libpcre2-posix.so.2 \ >>>>>> + /usr/lib/libpcre2-posix.so.2.0.3 \ >>>>>> + /usr/lib/libpng12.so \ >>>>>> + /usr/lib/libpng12.so.0 \ >>>>>> + /usr/lib/libpng12.so.0.57.0 \ >>>>>> + /usr/lib/libpng.so.3 \ >>>>>> + /usr/lib/libpng.so.3.57.0 \ >>>>>> + /usr/lib/libpoppler.so.100 \ >>>>>> + /usr/lib/libpoppler.so.100.0.0 \ >>>>>> + /usr/lib/libpoppler.so.110 \ >>>>>> + /usr/lib/libpoppler.so.110.0.0 \ >>>>>> + /usr/lib/libpoppler.so.111 \ >>>>>> + /usr/lib/libpoppler.so.111.0.0 \ >>>>>> + /usr/lib/libpoppler.so.66 \ >>>>>> + /usr/lib/libpoppler.so.66.0.0 \ >>>>>> + /usr/lib/libqpdf.so.17 \ >>>>>> + /usr/lib/libqpdf.so.17.0.0 \ >>>>>> + /usr/lib/libreadline.so.6 \ >>>>>> + /usr/lib/libreadline.so.6.3 \ >>>>>> + /usr/lib/libsensors.so.4 \ >>>>>> + /usr/lib/libsensors.so.4.4.0 \ >>>>>> + /usr/lib/libsqlite3.so \ >>>>>> + /usr/lib/libthreadutil.so \ >>>>>> + /usr/lib/libthreadutil.so.6 \ >>>>>> + /usr/lib/libthreadutil.so.6.0.3 \ >>>>>> + /usr/lib/libulockmgr.so \ >>>>>> + /usr/lib/libulockmgr.so.1 \ >>>>>> + /usr/lib/libulockmgr.so.1.0.1 \ >>>>>> + /usr/lib/libupnp.so \ >>>>>> + /usr/lib/libuuid.so \ >>>>>> + /usr/lib/libxml2.so \ >>>>>> + /usr/lib/libxslt.so \ >>>>>> + /usr/lib/pango \ >>>>>> + /usr/lib/perl5/site_perl/5.30.0 \ >>>>>> + /usr/lib/perl5/site_perl/5.32.1/Net/IP.pm \ >>>>>> + /usr/lib/python3.8/ensurepip/_bundled/pip-19.2.3-py2.py3-none-any.wh= l \ >>>>>> + /usr/lib/python3.8/idlelib/Icons/idle.icns \ >>>>>> + /usr/lib/python3.8/lib2to3/Grammar3.8.1.final.0.pickle \ >>>>>> + /usr/lib/python3.8/lib2to3/PatternGrammar3.8.1.final.0.pickle \ >>>>> >>>>> Why can we not delete /usr/lib/python3.8 as a whole? This should not be= needed any more. >>>> >>>> Yes, good point. I was running the comparison based on Core Update 164, = but with >>>> Python being updated to 3.10.x in Core Update 165, this folder won't be = necessary >>>> anymore completely. >>>> >>>>> >>>>>> + /usr/lib/sqlite3.34.0 \ >>>>>> + /usr/lib/squid/basic_nis_auth \ >>>>>> + /usr/lib/squid/ext_time_quota_acl \ >>>>>> + /usr/lib/tcl8/8.4/platform-1.0.14.tm \ >>>>>> + /usr/lib/tcl8/8.4/platform-1.0.15.tm \ >>>>>> + /usr/lib/tcl8/8.5/msgcat-1.6.0.tm \ >>>>>> + /usr/lib/tcl8/8.5/tcltest-2.4.0.tm \ >>>>>> + /usr/lib/tcl8/8.6/http-2.8.9.tm \ >>>>>> + /usr/lib/tcl8/8.6/tdbc/sqlite3-1.0.4.tm \ >>>>>> + /usr/lib/tcl8/8.6/tdbc/sqlite3-1.1.2.tm \ >>>>> >>>>> Why do we even ship tcl? Is anything reliant on this? >>>> >>>> I don't know. >>>> >>>>>> + /usr/lib/tdbc1.1.2 \ >>>>>> + /usr/lib/tdbcmysql1.1.2 \ >>>>>> + /usr/lib/tdbcodbc1.1.2 \ >>>>>> + /usr/lib/tdbcpostgres1.1.2 \ >>>>>> + /usr/lib/thread2.8.6 \ >>>>>> + /usr/libexec/xtables-addons \ >>>>>> /usr/local/bin/ovpn-ccd-convert \ >>>>>> - /usr/local/bin/rebuild-initrd >>>>>> + /usr/local/bin/rebuild-initrd \ >>>>>> + /usr/local/bin/xt_geoip_build \ >>>>>> + /usr/local/bin/xt_geoip_update \ >>>>>> + /usr/sbin/batctl \ >>>>>> + /usr/sbin/fbset \ >>>>>> + /usr/sbin/update-usbids.sh \ >>>>>> + /usr/sbin/uuidd \ >>>>>> + /usr/share/doc/fireinfo \ >>>>>> + /usr/share/GeoIP \ >>>>>> + /usr/share/zoneinfo/posix/US/Pacific-New \ >>>>>> + /usr/share/zoneinfo/right/US/Pacific-New \ >>>>>> + /usr/share/zoneinfo/US/Pacific-New \ >>>>>> + /var/lib/GeoIP >>>>> >>>>> What method did you use to identify all these files? >>>> >>>> On the IPFire installation in question, I ran >>>> >>>>> find bin/ etc/ lib/ opt/ run/ sbin/ srv/ usr/ var/ | sort > ~/ist >>>> >>>> and scp'ed that file onto my workstation. Then, I downloaded the ISO for= Core Update 164 >>>> (because I wanted to have something stable, which is why I did not choos= e C165 for this), >>>> installed it in a VM, rebooted it, mounted the root partition and execut= ed the same find >>>> command with a different file name. >>> >>> Okay, so this is based on a random system. Not bad, but would be better i= f we could diff two filelists from two core updates, wouldn=E2=80=99t it? >> >> I am not sure if I understood you here: The filelist above contains all le= ftovers that have >> agglomerated since Core Update 110. Comparing two filelists would only cov= er a fraction of >> it, since this comparison cannot skip any update. >> >> However: Do you see a feasible way to obtain such lists, and perhaps scipt= the generation of >> such a list of orphaned files? >=20 > Yes, we could use =E2=80=9Cgit archive=E2=80=9D to create a tarball from an= y branch, so in this case a core update from the past. >=20 > We could then only extract (or directly send to stdout) all files in config= /rootfiles/command, sort the output and compare it to what is currently in co= nfig/rootfiles/common. Stretch goal is doing this for all architectures :) >=20 > Here is the command for the main file list: >=20 > git archive origin/core160 | tar --extract --to-stdout config/rootfiles/c= ommon | grep -v "^#" | sort -u >=20 > This could be integrated into make.sh :) >=20 > -Michael >=20 >> >> Thanks, and best regards, >> Peter M=C3=BCller >> >>> >>>> >>>> A >>>> >>>>> diff -Naur soll ist | less >>>> >>>> (sorry for the German) outputs a long list, which I vetted manually sinc= e some add-ons >>>> were installed on the productive system, and I did not want to pipe the = output into the >>>> rm command blindly. >>>> >>>> Thanks, and best regards, >>>> Peter M=C3=BCller >>>> >>>>> >>>>> -Michael >>>>> >>>>>> >>>>>> # Delete old 2007 Pakfire key from GPG keyring >>>>>> export GNUPGHOME=3D"/opt/pakfire/etc/.gnupg" >>>>>> --=20 >>>>>> 2.34.1 >=20 --===============3784102029282667662==--