From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.haj.ipfire.org (localhost [IPv6:::1]) by mail02.haj.ipfire.org (Postfix) with ESMTP id 4jDSsl5PW6z2xXH for ; Tue, 27 Oct 2026 11:28:11 +0000 (UTC) Received: from mail01.ipfire.org (mail01.haj.ipfire.org [172.28.1.202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client CN "mail01.haj.ipfire.org", Issuer "YR1" (not verified)) by mail02.haj.ipfire.org (Postfix) with ESMTPS id 4jDSsl53lJz2xN3 for ; Tue, 27 Oct 2026 11:28:11 +0000 (UTC) Received: from people01.haj.ipfire.org (people01.haj.ipfire.org [IPv6:2001:678:b28::161]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (Client CN "people01.haj.ipfire.org", Issuer "YE1" (not verified)) by mail01.ipfire.org (Postfix) with ESMTPS id 4jDSsj59Jrz47J for ; Tue, 27 Oct 2026 11:28:09 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003ed25519; t=1793100489; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc; bh=PsvXieVU2CsJm+uPnm1wbfiz0b7VtUlW2N+6cTfL5hY=; b=cBFSP9YNisFdw9+R9U8lAvA8MZuieMJ0BVSqaYYaavsfZz73aN3HhBmLu7qPapf91MAQNb ekYzi1l2xK6nUNDQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfire.org; s=202003rsa; t=1793100489; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc; bh=PsvXieVU2CsJm+uPnm1wbfiz0b7VtUlW2N+6cTfL5hY=; b=Ej4iLXdcE7EU5PsaQB50mRUJIANGS6dvACpNyxv3izC7xfT/+6pSQ+1yJv4zG6FOCDHlyd Ej+MmPZ5RhWjhzdn/KdZfFP3787BwheNRqvHetc4QzWm3aZ6Ss6lCY9Mf+lyiYQ1iPWeza dkA/1w/QozTq8+ovhQQ9WI1fR5o1pyWpQjMlyJFqG+y154K0NuqmkQ6NLZK0MJdf4/k8se R6X66rXToWooUOZfgm6RdsYaC5pmnOKEbzyrY7Vu1OYfx02QKHNidvwFOL1jq6Wz3Cyup0 YwEm3jl9VSl8IRzzjSfdOa9tExNk81nFI9oZDIP6Sgxdq/cHloxn3ZC8sGUzbg== Received: by people01.haj.ipfire.org (Postfix, from userid 1000) id 4jDSsh3pjyz2xRB; Tue, 27 Oct 2026 11:28:08 +0000 (UTC) To: ipfire-scm@lists.ipfire.org Subject: [git.ipfire.org] IPFire 2.x development tree branch, next, updated. 0a32e2f01a0942dcd5a3012a4f9ecf9c0edcd30d X-Git-Refname: refs/heads/next X-Git-Reftype: branch X-Git-Oldrev: fe4c0b6931a8d586025885c8d0cb0dd7aa259b95 X-Git-Newrev: 0a32e2f01a0942dcd5a3012a4f9ecf9c0edcd30d Message-Id: <4jDSsh3pjyz2xRB@people01.haj.ipfire.org> Date: Tue, 27 Oct 2026 11:28:08 +0000 (UTC) From: Michael Tremer Precedence: list List-Id: List-Subscribe: , List-Unsubscribe: , List-Post: List-Help: Sender: Mail-Followup-To: This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "IPFire 2.x development tree". The branch, next has been updated via 0a32e2f01a0942dcd5a3012a4f9ecf9c0edcd30d (commit) via 90ae3cf49a45885ab4c1c96c49bbaca8dc608734 (commit) via dae7967fd2ac9c4a52719308531adcc63fe92784 (commit) via 870e5f896e75a06cd52ba22d42f9bb94cd062bb8 (commit) via 5f082c8b07a8e512faecba1cfc93dcb8aa362f7a (commit) via 3226836db6ebbf925c4133cfd71408676213102f (commit) via bd00153ebbf6e6c95f52c0ed9cab7459e42b41ae (commit) via b4327c047e9f80d02ac863345aae59a820548c48 (commit) via 328a43fcfebd224629c966a24c45b9cf30cdbc5b (commit) via b70831ff9b7556f5edc23cf1150f531addffca7a (commit) via ef178ec35f72c6fa2953b5bc14da3daa42947ab3 (commit) via 43d59542843a2f0c0f946de3b3ced97a3d19f07f (commit) via 8addb36acb4abdd0b3a82f2e4fafbaec486d9e18 (commit) via 3bfac46c93fb2b6038f8c798405035d692dac8aa (commit) via 0ecf0e2b8ab85c065d479c385cf6ea6d617edfef (commit) via 21d8d55713d5f0832f0249e2160fca4e38f7479d (commit) via 24ed1b6beb1843519af1d81c5dba64097f2ce3b5 (commit) via fe968ea54f5aa89e7698fd302ccf781ad0379739 (commit) via 483195399bf8cc8ca59e36adb81e35f0645ef1b3 (commit) via 6db8997f1a2c7839bc76bb174ebacdb380f082b1 (commit) via 6e106ecf1b41f456c70b4ab0ee50a9367b71f40c (commit) via 99bae9cd267b5d7af270676623af48934f8a9374 (commit) via ce44c0da81340719ad203f7b01cee786e6fde523 (commit) from fe4c0b6931a8d586025885c8d0cb0dd7aa259b95 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 0a32e2f01a0942dcd5a3012a4f9ecf9c0edcd30d Merge: fe4c0b693 90ae3cf49 Author: Michael Tremer Date: Tue Oct 27 11:28:02 2026 +0000 Merge branch 'pakfire-disk-space' into next commit 90ae3cf49a45885ab4c1c96c49bbaca8dc608734 Author: Michael Tremer Date: Tue Oct 27 11:18:33 2026 +0000 web-user-interface: Don't ask for a restart while pakfire is running Core Updates flag that a restart is required as soon as they have been installed. The web user interface showed this right away, although pakfire might still be busy for a while, for example with add-ons. The notice is now only shown once pakfire has finished. Fixes: #13149 Signed-off-by: Michael Tremer commit dae7967fd2ac9c4a52719308531adcc63fe92784 Author: Michael Tremer Date: Tue Oct 27 11:18:21 2026 +0000 pakfire: Wait for pakfire to finish before shutting down Core Updates tell the user that a reboot is required as soon as they have been installed, even if pakfire is still busy with add-ons. If the system was restarted then, pakfire was killed in the middle of its work. The pakfire initscript now has a stop action that is run first when the system is shut down or rebooted. If pakfire is running, it waits until pakfire has released its lock, for at most 30 minutes, so a pakfire waiting at an interactive prompt cannot block a reboot forever. Fixes: #13149 Signed-off-by: Michael Tremer commit 870e5f896e75a06cd52ba22d42f9bb94cd062bb8 Author: Michael Tremer Date: Tue Oct 27 11:11:02 2026 +0000 pakfire: Tidy up the package cache Downloaded add-on packages stayed in the cache forever. Their file names include the version, so every upgrade left another file behind, slowly filling up the disk. Packages are now removed from the cache once they have been installed, upgraded or removed, just like Core Updates. "pakfire upgrade", which also runs from cron, removes anything else that has been left behind. Signed-off-by: Michael Tremer commit 5f082c8b07a8e512faecba1cfc93dcb8aa362f7a Author: Michael Tremer Date: Tue Oct 27 11:10:19 2026 +0000 pakfire: Don't eval the core list coredbinfo() fetched the latest release by grepping the core list and running the result as Perl code. The list is signed, but there is no reason to execute anything from it. The release number is now parsed from the file. Signed-off-by: Michael Tremer commit 3226836db6ebbf925c4133cfd71408676213102f Author: Michael Tremer Date: Tue Oct 27 11:09:56 2026 +0000 pakfire: Move slow downloads to another mirror Mirrors are picked at random from the list, so a download could end up on a mirror that is very far away or very slow, and there was no way to move it elsewhere. pakfire now measures the speed of each download over 15 seconds. If it stays below 128 KiB/s, the download is stopped and continues on another mirror from where it stopped. If the connection itself is slow, every mirror will look slow. So the last attempt accepts any speed, and so does a download from a custom mirror or the main server, where there is nowhere else to go. Signed-off-by: Michael Tremer commit bd00153ebbf6e6c95f52c0ed9cab7459e42b41ae Author: Michael Tremer Date: Tue Oct 27 11:07:31 2026 +0000 pakfire: Continue interrupted downloads with the next mirror When a download broke off, pakfire threw away what it had received and started again from the beginning, which is painful for Core Updates of a few hundred MiB. The partial file is now kept, and the next attempt asks the next mirror for the rest only. If the mirror sends the whole file instead, it is written from the start. The signature is still checked on the complete file, so a file put together from two mirrors that don't match is rejected and downloaded again. A download that ends before the announced size has been received is now also treated as a failed attempt. Signed-off-by: Michael Tremer commit b4327c047e9f80d02ac863345aae59a820548c48 Author: Michael Tremer Date: Tue Oct 27 11:05:31 2026 +0000 pakfire: Try another mirror when a download fails When a download failed, pakfire picked another random mirror, which could easily be the same one again. A file that could not be verified made pakfire give up right away, so a single broken mirror could stop an update. Mirrors that have failed are now avoided for the rest of that download, and a file that does not verify is treated like any other failed attempt. pakfire only gives up, and suggests checking the clock, if none of the attempts produced a valid file. selectmirror() also tried to read the server list after downloading it for the first time without opening it again, and fell back to the main server. It now reads the list properly. Signed-off-by: Michael Tremer commit 328a43fcfebd224629c966a24c45b9cf30cdbc5b Author: Michael Tremer Date: Tue Oct 27 11:02:45 2026 +0000 pakfire: Move the lockfile to /run The lockfile was in /tmp, where anyone can create files, so another user could create it first or replace it with a symlink. /run is only writable by root and is emptied on every boot. Signed-off-by: Michael Tremer commit b70831ff9b7556f5edc23cf1150f531addffca7a Author: Michael Tremer Date: Tue Oct 27 10:58:22 2026 +0000 pakfire: Show all packages in "pakfire info" if one is unknown "pakfire info" stopped at the first unknown package and did not show any of the packages after it. Signed-off-by: Michael Tremer commit ef178ec35f72c6fa2953b5bc14da3daa42947ab3 Author: Michael Tremer Date: Tue Oct 27 10:58:17 2026 +0000 pakfire: Fix inverted check after failing to download the package list If the package list could not be downloaded, dbgetlist() was supposed to give up only when there was no list at all. The check was inverted: it gave up when an older list existed, so metadata wasn't refreshed, and carried on without any list. Signed-off-by: Michael Tremer commit 43d59542843a2f0c0f946de3b3ced97a3d19f07f Author: Michael Tremer Date: Tue Oct 27 10:58:07 2026 +0000 pakfire: Report package updates in the exit code of "pakfire status" "pakfire status" should add 3 to its exit code if there are package updates available. It compared the number of available updates to "yes", so this never happened. Signed-off-by: Michael Tremer commit 8addb36acb4abdd0b3a82f2e4fafbaec486d9e18 Author: Michael Tremer Date: Tue Oct 27 10:58:02 2026 +0000 pakfire.cgi: Don't consider pakfire busy because of a leftover lockfile The web user interface treated pakfire as running whenever the lockfile existed. A file left behind by a killed instance blocked installing, removing and upgrading from the web user interface. The new Pakfire::is_locked() checks whether a running pakfire actually holds the lock instead. Signed-off-by: Michael Tremer commit 3bfac46c93fb2b6038f8c798405035d692dac8aa Author: Michael Tremer Date: Tue Oct 27 10:57:31 2026 +0000 pakfire: Use a real lock instead of only checking for the lockfile If pakfire was killed, its lockfile was left behind and pakfire refused to run again until the file was removed by hand. pakfire now holds an flock() on the lockfile, which the kernel releases when the process exits in any way. A leftover file no longer blocks anything. The file is still created and removed as before, because the web user interface uses it to show that pakfire is running. The lockfile is in /tmp, so it is now opened without following symlinks. Signed-off-by: Michael Tremer commit 0ecf0e2b8ab85c065d479c385cf6ea6d617edfef Author: Michael Tremer Date: Tue Oct 27 10:56:52 2026 +0000 pakfire: Only download over HTTPS pakfire still accepted mirrors with plain HTTP and followed redirects from HTTPS to HTTP. All mirrors use HTTPS today, and a custom mirror is already required to use HTTPS. HTTP mirrors are now skipped, and the user agent refuses anything other than HTTPS, including redirects. Signed-off-by: Michael Tremer commit 21d8d55713d5f0832f0249e2160fca4e38f7479d Author: Michael Tremer Date: Tue Oct 27 10:56:32 2026 +0000 pakfire: Don't accept the 2007 key any more The 2007 key was dropped and removed from all systems in Core Update 166 already, so its fingerprint should no longer be on the list of accepted keys either. Signed-off-by: Michael Tremer commit 24ed1b6beb1843519af1d81c5dba64097f2ce3b5 Author: Michael Tremer Date: Tue Oct 27 10:56:07 2026 +0000 pakfire: Accept packages signed with the 2022 key The 2022 key has been shipped and imported since Core Update 166, but pakfire never added its fingerprint to the list of accepted keys. Anything signed with it would have been rejected, so the key could not actually be used. Signed-off-by: Michael Tremer commit fe968ea54f5aa89e7698fd302ccf781ad0379739 Author: Michael Tremer Date: Tue Oct 27 10:52:24 2026 +0000 pakfire: Add a command to re-install the current Core Update Testers sometimes need to install a Core Update again, for example after it has been rebuilt on the testing tree. So far, this was only done implicitly by "pakfire upgrade" when a newer release came out. "pakfire reinstall-core" now downloads the installed release again and re-installs it. It is only available on the testing and unstable trees. Because rebuilt updates keep their file name, cached metadata and packages are removed first so that the latest build is downloaded. "pakfire upgrade" does the same for the release it re-installs. Signed-off-by: Michael Tremer commit 483195399bf8cc8ca59e36adb81e35f0645ef1b3 Author: Michael Tremer Date: Tue Oct 27 10:51:16 2026 +0000 pakfire: Don't pass log messages through a shell logger() called the logger command through a shell with the message in double quotes. Anything like $(...) or `...` in a message was executed as root. Log messages include data from outside, for example the HTTP status line that a mirror sends and package names that are passed on the command line or from the web UI. A mirror could therefore run commands on any system that downloads from it, regardless of package signatures. logger is now called directly without a shell. Signed-off-by: Michael Tremer commit 6db8997f1a2c7839bc76bb174ebacdb380f082b1 Author: Michael Tremer Date: Tue Oct 27 10:43:33 2026 +0000 pakfire: Install Core Updates one after the other When a system was more than one Core Update behind, pakfire downloaded all updates before installing the first one. The disk needed space for all of them at the same time, and they stayed in the cache afterwards. Now each update is downloaded, installed and removed from the cache before the next one is fetched. The installed release is recorded after each step, so an interrupted upgrade continues from where it stopped. Signed-off-by: Michael Tremer commit 6e106ecf1b41f456c70b4ab0ee50a9367b71f40c Author: Michael Tremer Date: Tue Oct 27 10:43:21 2026 +0000 pakfire: Check for free disk space before extracting a package If the disk fills up while a package is being extracted, the system is left half-updated, which is especially bad for Core Updates. After decrypting a package, pakfire now lists the archive and works out how much the files will add on disk, counting existing files only if they grow. If that does not fit, pakfire stops before the install or update script has changed anything. Signed-off-by: Michael Tremer commit 99bae9cd267b5d7af270676623af48934f8a9374 Author: Michael Tremer Date: Tue Oct 27 10:42:45 2026 +0000 pakfire: Check for free disk space before downloading pakfire never checked whether there was enough space on disk before it started downloading packages. It would simply fill up the disk and fail somewhere in the middle. Before downloading anything, pakfire now checks whether all packages fit into the cache and whether there is enough space left to decrypt the largest of them. fetchfile() also checks the size that the server announces before it starts writing. 64 MiB are always kept free so that the system keeps running. Signed-off-by: Michael Tremer commit ce44c0da81340719ad203f7b01cee786e6fde523 Author: Michael Tremer Date: Tue Oct 27 10:42:13 2026 +0000 pakfire: Write downloads straight to disk So far, pakfire kept the entire download in memory before writing it to disk. With Core Updates of a few hundred MiB, this could exhaust the memory of smaller systems. The file is now written while it is being received. If writing fails, for example because the disk is full, the download is aborted and the partial file is removed. Signed-off-by: Michael Tremer ----------------------------------------------------------------------- Summary of changes: config/rootfiles/common/initscripts | 2 + html/cgi-bin/index.cgi | 3 +- html/cgi-bin/pakfire.cgi | 7 +- lfs/initscripts | 2 + src/initscripts/system/pakfire | 17 +- src/pakfire/lib/functions.pl | 545 ++++++++++++++++++++++++++++++------ src/pakfire/pakfire | 61 +++- 7 files changed, 540 insertions(+), 97 deletions(-) Difference in files: diff --git a/config/rootfiles/common/initscripts b/config/rootfiles/common/initscripts index e0f182826..460d68c09 100644 --- a/config/rootfiles/common/initscripts +++ b/config/rootfiles/common/initscripts @@ -101,6 +101,7 @@ etc/rc.d/init.d/waitfordns etc/rc.d/init.d/wireguard etc/rc.d/init.d/wlanclient #etc/rc.d/rc0.d +etc/rc.d/rc0.d/K00pakfire etc/rc.d/rc0.d/K01grub-btrfsd #etc/rc.d/rc0.d/K01vdradmin etc/rc.d/rc0.d/K08fcron @@ -158,6 +159,7 @@ etc/rc.d/rc3.d/S98rc.local etc/rc.d/rc3.d/S99grub-btrfsd #etc/rc.d/rc3.d/S99vdradmin #etc/rc.d/rc6.d +etc/rc.d/rc6.d/K00pakfire etc/rc.d/rc6.d/K01grub-btrfsd #etc/rc.d/rc6.d/K01vdradmin etc/rc.d/rc6.d/K08fcron diff --git a/html/cgi-bin/index.cgi b/html/cgi-bin/index.cgi index 0f51ba9ff..bd13cff55 100644 --- a/html/cgi-bin/index.cgi +++ b/html/cgi-bin/index.cgi @@ -595,7 +595,8 @@ if (defined $coredb{'AvailableRelease'}) { push(@warnings, "@{[ sprintf(_('Notice: There is a core update from %s to %s available.'), $coredb{'Release'}, $coredb{'AvailableRelease'}) ]}"); } -if ( -e "/var/run/need_reboot" ) { +# Don't ask for a restart while pakfire is still running +if ( -e "/var/run/need_reboot" && !&Pakfire::is_locked() ) { push(@warnings, _('An Update Requires a Restart')); } diff --git a/html/cgi-bin/pakfire.cgi b/html/cgi-bin/pakfire.cgi index 08381fc4b..2974d01aa 100644 --- a/html/cgi-bin/pakfire.cgi +++ b/html/cgi-bin/pakfire.cgi @@ -715,7 +715,8 @@ $selected{"TREE"}{$pakfiresettings{"TREE"}} = "selected"; print < END -if ($pakfire_status{'RebootRequired'} eq "yes") +# Don't ask for a restart while pakfire is still running +if (($pakfire_status{'RebootRequired'} eq "yes") && !&_is_pakfire_busy()) { print < @@ -868,8 +869,8 @@ END # Check if pakfire is already running (extend test here if necessary) sub _is_pakfire_busy { - # Return immediately if lockfile is present - if(-e "$Pakfire::lockfile") { + # Return immediately if another instance holds the lock + if(&Pakfire::is_locked()) { return 1; } diff --git a/lfs/initscripts b/lfs/initscripts index fd361a707..775e3115a 100644 --- a/lfs/initscripts +++ b/lfs/initscripts @@ -91,6 +91,7 @@ $(TARGET) : install -v -m 755 $$i /etc/rc.d/helper/; \ done + ln -sf ../init.d/pakfire /etc/rc.d/rc0.d/K00pakfire ln -sf ../init.d/grub-btrfsd /etc/rc.d/rc0.d/K01grub-btrfsd ln -sf ../init.d/vdradmin /etc/rc.d/rc0.d/K01vdradmin ln -sf ../init.d/fcron /etc/rc.d/rc0.d/K08fcron @@ -146,6 +147,7 @@ $(TARGET) : ln -sf ../init.d/grub-btrfsd /etc/rc.d/rc3.d/S99grub-btrfsd ln -sf ../init.d/vdradmin /etc/rc.d/rc3.d/S99vdradmin + ln -sf ../init.d/pakfire /etc/rc.d/rc6.d/K00pakfire ln -sf ../init.d/grub-btrfsd /etc/rc.d/rc6.d/K01grub-btrfsd ln -sf ../init.d/vdradmin /etc/rc.d/rc6.d/K01vdradmin ln -sf ../init.d/fcron /etc/rc.d/rc6.d/K08fcron diff --git a/src/initscripts/system/pakfire b/src/initscripts/system/pakfire index 9b3c043e9..c86ca00c7 100644 --- a/src/initscripts/system/pakfire +++ b/src/initscripts/system/pakfire @@ -24,6 +24,12 @@ export GNUPGHOME="/opt/pakfire/etc/.gnupg" +# The lock that is held while pakfire is running +LOCKFILE="/run/pakfire.lock" + +# Don't wait longer than this (in seconds) when shutting down +TIMEOUT=1800 + case "${1}" in start) boot_mesg "Setting up Pakfire Package Manager..." @@ -31,8 +37,17 @@ case "${1}" in evaluate_retval ;; + stop) + # Wait for a running instance of pakfire to finish before shutting down + if [ -e "${LOCKFILE}" ] && ! flock --nonblock --shared "${LOCKFILE}" true; then + boot_mesg "Waiting for Pakfire to finish..." + flock --timeout "${TIMEOUT}" --shared "${LOCKFILE}" true + evaluate_retval + fi + ;; + *) - echo "Usage: ${0} {start}" + echo "Usage: ${0} {start|stop}" exit 1 ;; esac diff --git a/src/pakfire/lib/functions.pl b/src/pakfire/lib/functions.pl index e2c646313..9f9fe5a63 100644 --- a/src/pakfire/lib/functions.pl +++ b/src/pakfire/lib/functions.pl @@ -24,6 +24,8 @@ require "/var/ipfire/general-functions.pl"; use File::Basename; use File::Copy; +use Fcntl qw(:flock); +use Time::HiRes; use LWP::UserAgent; use HTTP::Response; use HTTP::Headers; @@ -37,10 +39,10 @@ use Switch; package Pakfire; my @VALID_KEY_FINGERPRINTS = ( + # 2022 + "98448779295007D302A91E32A551AE95C8DCE211", # 2018 "3ECA8AA4478208B924BB96206FEF7A8ED713594B", - # 2007 - "179740DC4D8C47DC63C099C74BDE364C64D96617", ); # A small color-hash :D @@ -64,7 +66,14 @@ our %color; $color{'white'} = "\033[1;37m"; our $enable_colors = 1; -my $final_data; +my $download_fh; +my $download_size; +my $download_error; +my $download_offset; +my $download_min_speed; +my $download_too_slow; +my $download_window_start; +my $download_window_size; my $total_size; my $bfile; @@ -75,7 +84,28 @@ my %pakfiresettings = (); $Conf::version = &make_version(); # Pakfire lock file. -our $lockfile = "/tmp/pakfire_lock"; +our $lockfile = "/run/pakfire.lock"; + +# Always keep this much space free on disk (in bytes) +our $space_reserve = 64 * 1024 * 1024; + +# Downloads that are slower than this (in bytes per second) are moved to another mirror +our $min_download_speed = 128 * 1024; + +# The download speed is measured over this many seconds +our $download_speed_interval = 15; + +sub is_locked { + ### This subroutine returns 1 if a running instance of pakfire holds the lock. + # A lockfile that has been left behind by a killed instance is ignored. + open(my $fh, "<", "$lockfile") or return 0; + + # We can't get a shared lock while pakfire is holding its exclusive lock + my $locked = !flock($fh, Fcntl::LOCK_SH | Fcntl::LOCK_NB); + close($fh); + + return $locked ? 1 : 0; +} sub message { my $message = shift; @@ -105,7 +135,8 @@ sub message { sub logger { my $log = shift; if ($log) { - system("logger -t pakfire \"$log\""); + # Don't pass the message through a shell + system("/usr/bin/logger", "-t", "pakfire", "--", "$log"); } } @@ -119,6 +150,9 @@ sub usage { &Pakfire::message(" remove - remove one or more packages"); &Pakfire::message(" update - update the pakfire database"); &Pakfire::message(" upgrade - upgrade all packages to the latest version"); + &Pakfire::message(" reinstall-core"); + &Pakfire::message(" - download and re-install the current Core Update"); + &Pakfire::message(" (only on the testing and unstable trees)"); &Pakfire::message(" list [installed | notinstalled | upgrade]"); &Pakfire::message(" - display a list of installed, notinstalled,"); &Pakfire::message(" upgradeable or all packages"); @@ -141,24 +175,34 @@ sub fetchfile { my $gethost = shift; my (@server, $host, $proto, $file, $i); my $allok = 0; + my $switchable = 0; + + # Hosts that failed to deliver the file + my %failed; + my $signature_failed = 0; use File::Basename; $bfile = basename("$getfile"); logger("DOWNLOAD STARTED: $getfile"); + # Don't continue anything that might have been left behind + unlink("$Conf::tmpdir/$bfile"); + $i = 0; while (($allok == 0) && $i < 5) { $i++; if ("$gethost" eq "") { - @server = selectmirror(); + @server = selectmirror(\%failed); $proto = $server[0]; $host = $server[1]; $file = "$server[2]/$getfile"; + $switchable = $server[3]; } else { $host = $gethost; $file = $getfile; + $switchable = 0; } $proto = "HTTPS" unless $proto; @@ -174,6 +218,9 @@ sub fetchfile { } ); $ua->agent("Pakfire/$Conf::version"); + + # Only use HTTPS, also when following redirects + $ua->protocols_allowed(["https"]); $ua->timeout(20); my %proxysettings=(); @@ -189,11 +236,8 @@ sub fetchfile { } } - $final_data = undef; - my $url; switch ($proto) { - case "HTTP" { $url = "http://$host/$file"; } case "HTTPS" { $url = "https://$host/$file"; } else { # skip all lines with unknown protocols @@ -207,38 +251,107 @@ sub fetchfile { $total_size = $remote_headers->content_length; logger("DOWNLOAD INFO: $file has size of $total_size bytes"); - my $response = $ua->get($url, ':content_cb' => \&callback ); + # Continue where a previous attempt has stopped + $download_offset = (-s "$Conf::tmpdir/$bfile") || 0; + + if ($total_size && $download_offset >= $total_size) { + unlink("$Conf::tmpdir/$bfile"); + $download_offset = 0; + } + + # Don't start the download if it won't fit on disk + if ($total_size && !&check_free_space($Conf::tmpdir, $total_size - $download_offset, "download $bfile")) { + exit 1; + } + + my @headers; + if ($download_offset) { + logger("DOWNLOAD INFO: Continuing $bfile at byte $download_offset"); + push(@headers, "Range" => "bytes=$download_offset-"); + } + + # Move on to another mirror if this one is too slow, but accept any speed + # on the last attempt and if there is no other mirror to go to + $download_min_speed = ($switchable && $i < 5) ? $min_download_speed : 0; + $download_too_slow = undef; + + # The file is opened when the first data arrives + $download_fh = undef; + $download_size = $download_offset; + $download_error = undef; + + my $response = $ua->get($url, @headers, ':content_cb' => \&callback ); message(""); + if ($download_fh) { + unless (close($download_fh)) { + $download_error //= "$!"; + } + $download_fh = undef; + } + my $code = $response->code(); my $log = $response->status_line; logger("DOWNLOAD INFO: HTTP-Status-Code: $code - $log"); - if ($response->is_success) { - if (open(FILE, ">$Conf::tmpdir/$bfile")) { - print FILE $final_data; - close(FILE); - logger("DOWNLOAD INFO: File received. Start checking signature..."); - if (&valid_signature("$Conf::tmpdir/$bfile")) { - logger("DOWNLOAD INFO: Signature of $bfile is fine."); - move("$Conf::tmpdir/$bfile","$Conf::cachedir/$bfile"); - } else { - message("DOWNLOAD ERROR: The downloaded file ($file) wasn't verified by IPFire.org. Sorry - Exiting..."); - my $ntp = `ntpdate -q -t 10 pool.ntp.org 2>/dev/null | tail -1`; - if ( $ntp !~ /time\ server(.*)offset(.*)/ ){message("TIME ERROR: Unable to get the nettime, this may lead to the verification error.");} - else { $ntp =~ /time\ server(.*)offset(.*)/; message("TIME INFO: Time Server$1has$2 offset to localtime.");} - exit 1; - } - logger("DOWNLOAD FINISHED: $file"); - $allok = 1; - return 1; - } else { - logger("DOWNLOAD ERROR: Could not open $Conf::tmpdir/$bfile for writing."); + # Abort if we could not write the file (e.g. because the disk is full) + if (defined $download_error) { + unlink("$Conf::tmpdir/$bfile"); + message("DOWNLOAD ERROR: Could not write $Conf::tmpdir/$bfile: $download_error"); + exit 1; + } + + # Keep what we have and continue with another mirror if this one was too slow + if (defined $download_too_slow) { + message("DOWNLOAD WARN: $host is too slow (" . beautifysize($download_too_slow) + . "/s). Trying another mirror..."); + $failed{$host} = 1; + next; + } + + if ($response->is_success && !$response->header("X-Died")) { + # Continue with another mirror if the connection was closed too early + if ($total_size && ((-s "$Conf::tmpdir/$bfile") || 0) < $total_size) { + logger("DOWNLOAD ERROR: $host: Received an incomplete file"); + $failed{$host} = 1; + next; + } + + logger("DOWNLOAD INFO: File received. Start checking signature..."); + unless (&valid_signature("$Conf::tmpdir/$bfile")) { + # Try another mirror + message("DOWNLOAD WARN: The file received from $host could not be verified. Trying another mirror..."); + unlink("$Conf::tmpdir/$bfile"); + $failed{$host} = 1; + $signature_failed = 1; + next; } + + logger("DOWNLOAD INFO: Signature of $bfile is fine."); + move("$Conf::tmpdir/$bfile","$Conf::cachedir/$bfile"); + + logger("DOWNLOAD FINISHED: $file"); + $allok = 1; + return 1; } else { - logger("DOWNLOAD ERROR: $log"); + # Keep what we have received so far so that the next mirror can continue + # unless the server told us that the requested range is invalid + unlink("$Conf::tmpdir/$bfile") if ($code == 416); + + logger("DOWNLOAD ERROR: $host: $log"); + $failed{$host} = 1; } } + + # If no file could be verified, the local clock might be wrong + if ($signature_failed) { + message("DOWNLOAD ERROR: The downloaded file ($getfile) wasn't verified by IPFire.org. Sorry - Exiting..."); + my $ntp = `ntpdate -q -t 10 pool.ntp.org 2>/dev/null | tail -1`; + if ( $ntp !~ /time\ server(.*)offset(.*)/ ){message("TIME ERROR: Unable to get the nettime, this may lead to the verification error.");} + else { $ntp =~ /time\ server(.*)offset(.*)/; message("TIME INFO: Time Server$1has$2 offset to localtime.");} + exit 1; + } + message("DOWNLOAD ERROR: There was no chance to get the file \"$getfile\" from any available server.\nMay be you should run \"pakfire update\" to get some new servers."); return 0; } @@ -316,12 +429,18 @@ sub valid_signature($) { } sub selectmirror { + ### This subroutine returns a randomly chosen mirror as (protocol, host, path, switchable). + # Mirrors whose host is in the (optional) hash of failed hosts are avoided + # unless there are no others left. switchable is 1 if there are other mirrors + # that could be used instead. + my $failed = shift || {}; + if (defined ${Conf::mirror}) { my $uri = URI->new("${Conf::mirror}"); # Only accept HTTPS mirrors if ($uri->scheme eq "https") { - return ("HTTPS", $uri->host, $uri->path . "/" . ${Conf::version}); + return ("HTTPS", $uri->host, $uri->path . "/" . ${Conf::version}, 0); } else { message("MIRROR ERROR: Unsupported mirror: " . ${Conf::mirror}); } @@ -329,53 +448,38 @@ sub selectmirror { ### Check if there is a current server list and read it. # If there is no list try to get one. - unless (open(FILE, "<$Conf::dbdir/lists/server-list.db")) { + unless ( -e "$Conf::dbdir/lists/server-list.db" ) { unless (getmirrors("noforce")) { message("MIRROR ERROR: Could not find or download a server list"); exit 1; } } - my @lines = ; - close(FILE); - - ### Count the number of the servers in the list - my $scount = 0; - my @newlines; - foreach (@lines) { - if ("$_" =~ /.*;.*;.*;/ ) { - push(@newlines,$_); - $scount++; - } + my @lines; + if (open(FILE, "<$Conf::dbdir/lists/server-list.db")) { + @lines = ; + close(FILE); } + + ### Collect all servers in the list + my @servers = grep { /.*;.*;.*;/ } @lines; + my $scount = scalar @servers; + logger("MIRROR INFO: $scount servers found in list"); - if ($scount eq 0) { + if ($scount == 0) { logger("MIRROR INFO: Could not find any servers. Falling back to main server $Conf::mainserver"); - return ("HTTPS", $Conf::mainserver, "/$Conf::version"); - } - - ### Choose a random server and test if it is online - # If the check fails try a new server. - # This will never give up. - my $servers = 0; - while (1) { - $server = int(rand($scount) + 1); - $servers = 0; - my ($line, $proto, $path, $host); - my @templine; - foreach $line (@newlines) { - $servers++; - if ($servers eq $server) { - @templine = split(/\;/, $line); - $proto = $templine[0]; - $host = $templine[1]; - $path = $templine[2]; - - return ($proto, $host, $path); - } - } + return ("HTTPS", $Conf::mainserver, "/$Conf::version", 0); } + + ### Prefer servers that have not failed yet + my @candidates = grep { my @t = split(/\;/, $_); !$failed->{$t[1]} } @servers; + @candidates = @servers unless (@candidates); + + ### Choose a random server + my @templine = split(/\;/, $candidates[int(rand(scalar @candidates))]); + + return ($templine[0], $templine[1], $templine[2], ($scount > 1) ? 1 : 0); } sub dbgetlist { @@ -401,7 +505,7 @@ sub dbgetlist { if ("$force" eq "force") { if (fetchfile("lists/packages_list.db", "")) { move("$Conf::cachedir/packages_list.db", "$Conf::dbdir/lists/packages_list.db"); - } elsif ( -e "$Conf::dbdir/lists/packages_list.db" ) { + } elsif (! -e "$Conf::dbdir/lists/packages_list.db" ) { # If we end up with no db file after download error there # is nothing more we can do here. return 0; @@ -441,7 +545,18 @@ sub coredbinfo { ### This subroutine returns core db version information in a hash. # Usage is without arguments - eval(`grep "core_" $Conf::dbdir/lists/core-list.db`); + # Read the latest release from the core list + $core_release = 0; + + if (open(my $fh, "<", "$Conf::dbdir/lists/core-list.db")) { + while (<$fh>) { + if (/^\$core_release\s*=\s*"(\d+)";/) { + $core_release = $1; + last; + } + } + close($fh); + } my %coredb = ( CoreVersion => $Conf::version, @@ -646,6 +761,28 @@ sub cleanup { } } +sub remove_from_cache { + ### This subroutine removes the downloaded file of a package from the cache. + my $pak = shift; + + return unless ( -e "$Conf::dbdir/meta/meta-$pak" ); + + my %metadata = parsemetafile("$Conf::dbdir/meta/meta-$pak"); + + if ($metadata{'File'}) { + unlink("$Conf::cachedir/$metadata{'File'}"); + } +} + +sub tidy_cache { + ### This subroutine removes all downloaded packages from the cache. + # They are only needed until they have been installed. + foreach my $file (glob("$Conf::cachedir/*.ipfire")) { + logger("CLEANUP: Removing $file"); + unlink($file); + } +} + sub getmetafile { my $pak = shift; @@ -815,12 +952,15 @@ sub setuppak { message("PAKFIRE INST: $pak: Decrypting..."); decryptpak("$pak"); + check_install_space("$pak"); + message("PAKFIRE INST: $pak: Copying files and running post-installation scripts..."); my $return = system("cd $Conf::tmpdir && NAME=$pak ./install.sh >> $Conf::logdir/install-$pak.log 2>&1"); $return %= 255; if ($return == 0) { move("$Conf::tmpdir/ROOTFILES", "$Conf::dbdir/rootfiles/$pak"); cleanup("tmp"); + remove_from_cache("$pak"); copy("$Conf::dbdir/meta/meta-$pak","$Conf::dbdir/installed/"); message("PAKFIRE INST: $pak: Finished."); message(""); @@ -842,18 +982,67 @@ sub upgradecore { my @seq = ($Conf::core_mine .. $core_release); shift @seq; - my $release; - foreach $release (@seq) { - chomp($release); - getpak("core-upgrade-$release"); + + # Download and install one update after the other so that we never + # need space for more than one of them at a time + foreach my $release (@seq) { + # Always fetch the release that is being re-installed again, + # because it might have been rebuilt in the meantime + my $force = ($release == $Conf::core_mine + 1 && $tree ne "stable") ? "force" : "noforce"; + + installcore($release, $force); } - - foreach $release (@seq) { - chomp($release); - upgradepak("core-upgrade-$release"); +} + +sub reinstallcore { + ### This subroutine downloads the currently installed Core Update again + # and re-installs it. This is only allowed for testers. + my $tree = &get_tree(); + + unless ($tree eq "testing" || $tree eq "unstable") { + message("CORE ERROR: Core Updates can only be re-installed on the testing or unstable tree."); + exit 1; + } + + unless ($Conf::core_mine > 0) { + message("CORE ERROR: Could not determine which Core Update is installed."); + exit 1; + } + + message("CORE UPGR: Re-installing release $Conf::core_mine"); + + installcore($Conf::core_mine, "force"); +} + +sub installcore($$) { + ### This subroutine downloads and installs a single Core Update. + # If force is passed, metadata and package are always downloaded again. + my ($release, $force) = @_; + my $pak = "core-upgrade-$release"; + + # Remove any cached package and metadata + if ("$force" eq "force") { + remove_from_cache($pak); + unlink("$Conf::dbdir/meta/meta-$pak"); + } + + unless (getmetafile($pak)) { + message("CORE ERROR: Could not retrieve the metadata for release $release."); + exit 1; + } + + unless (check_download_space($pak)) { + exit 1; + } + + getpak($pak); + upgradepak($pak); + + # Remember which release we are on now + if (open(my $fh, ">", "$Conf::coredir/mine")) { + print $fh "$release\n"; + close($fh); } - - system("echo $core_release > $Conf::coredir/mine"); } sub isinstalled { @@ -872,12 +1061,15 @@ sub upgradepak { message("PAKFIRE UPGR: $pak: Decrypting..."); decryptpak("$pak"); + check_install_space("$pak"); + message("PAKFIRE UPGR: $pak: Upgrading files and running post-upgrading scripts..."); my $return = system("cd $Conf::tmpdir && NAME=$pak ./update.sh >> $Conf::logdir/update-$pak.log 2>&1"); $return %= 255; if ($return == 0) { move("$Conf::tmpdir/ROOTFILES", "$Conf::dbdir/rootfiles/$pak"); cleanup("tmp"); + remove_from_cache("$pak"); copy("$Conf::dbdir/meta/meta-$pak", "$Conf::dbdir/installed/"); message("PAKFIRE UPGR: $pak: Finished."); message(""); @@ -901,6 +1093,7 @@ sub removepak { unlink("$Conf::dbdir/rootfiles/$pak"); unlink("$Conf::dbdir/installed/meta-$pak"); cleanup("tmp"); + remove_from_cache("$pak"); message("PAKFIRE REMV: $pak: Finished."); message(""); } else { @@ -929,6 +1122,148 @@ sub beautifysize { return $string; } +sub free_space($) { + ### This subroutine returns the number of bytes that are available + # on the file system that holds the given path. + # It returns undef if this could not be determined. + my $path = shift; + + open(my $cmd, "-|", "/bin/df", "--portability", "--block-size=1", $path) or return undef; + my @lines = <$cmd>; + close($cmd); + + return undef unless (@lines >= 2); + + # Filesystem, Size, Used, Available, Use%, Mounted on + my @fields = split(/\s+/, $lines[-1]); + + return undef unless ($fields[3] =~ /^\d+$/); + + return $fields[3]; +} + +sub check_free_space($$$) { + ### This subroutine checks if there are at least the given number of bytes + # (plus some reserve) available on the file system that holds the given + # path. It prints an error and returns 0 if that is not the case. + my ($path, $required, $action) = @_; + + my $available = &free_space($path); + + # Don't stop anything if we could not determine the free space + unless (defined $available) { + logger("PAKFIRE WARN: Could not determine the free space on $path"); + return 1; + } + + $required += $space_reserve; + + logger("PAKFIRE INFO: $action requires " . beautifysize($required) + . ", " . beautifysize($available) . " are available on $path"); + + if ($available < $required) { + message("PAKFIRE ERROR: There is not enough free disk space to $action."); + message("PAKFIRE ERROR: Required: " . beautifysize($required) + . ", available: " . beautifysize($available)); + return 0; + } + + return 1; +} + +sub download_size { + ### This subroutine returns the number of bytes that need to be downloaded + # for the given packages. Packages that are already in the cache are skipped. + my @paks = @_; + my $size = 0; + + foreach my $pak (@paks) { + next unless (getmetafile("$pak")); + + my %metadata = parsemetafile("$Conf::dbdir/meta/meta-$pak"); + + # Skip anything that we have already downloaded + next if ($metadata{'File'} && -e "$Conf::cachedir/$metadata{'File'}"); + + $size += $metadata{'Size'} || 0; + } + + return $size; +} + +sub check_download_space { + ### This subroutine checks if there is enough space to download all + # given packages and to decrypt the largest of them. + my @paks = @_; + + my $largest = 0; + foreach my $pak (@paks) { + my $size = &getsize("$pak") || 0; + + $largest = $size if ($size > $largest); + } + + return check_free_space($Conf::cachedir, &download_size(@paks) + $largest, + "download " . join(", ", @paks)); +} + +sub install_size() { + ### This subroutine returns how many bytes the files of the currently + # decrypted package will additionally occupy on disk once extracted. + # Files that already exist are only counted if they grow. + # It returns undef if this could not be determined. + my $block_size = 4096; + my $size = 0; + + my ($archive) = glob("$Conf::tmpdir/files*"); + return undef unless ($archive); + + open(my $cmd, "-|", "/bin/tar", "--list", "--verbose", "--numeric-owner", "--file", $archive) + or return undef; + + while (<$cmd>) { + chomp; + + # Only regular files take up space (hard links are listed as "h") + next unless (/^-\S*\s+\S+\s+(\d+)\s+\S+\s+\S+\s+(.+)$/); + + my $new_size = $1; + my $path = "/$2"; + + # Round up to whole blocks + my $new_blocks = int(($new_size + $block_size - 1) / $block_size); + my $old_blocks = 0; + + if (-f $path && ! -l $path) { + $old_blocks = int(((-s $path || 0) + $block_size - 1) / $block_size); + } + + $size += ($new_blocks - $old_blocks) * $block_size if ($new_blocks > $old_blocks); + } + + close($cmd) or return undef; + + return $size; +} + +sub check_install_space { + ### This subroutine checks if there is enough space to extract the + # currently decrypted package and exits if that is not the case. + my $pak = shift; + + my $size = &install_size(); + + unless (defined $size) { + logger("PAKFIRE WARN: Could not determine the installed size of $pak"); + return; + } + + unless (check_free_space("/", $size, "install $pak")) { + cleanup("tmp"); + exit 1; + } +} + sub makeuuid { unless ( -e "$Conf::dbdir/uuid" ) { open(FILE, "$Pakfire::lockfile"); + # Update the timestamp of the lockfile (shown in the web user interface) + utime(undef, undef, "$Pakfire::lockfile"); # Pakfire has locked in this session set locket to "1". + # The lockfile stays open until pakfire exits. $locked = "1"; - # Close filehandle. - close(LOCK); - ### Check if we are started by another name # if ( $0 =~ /pakfire-update$/ ) { @@ -188,6 +195,11 @@ # my %sort = map{ $_, 1 } @all; # @all = sort keys %sort; + ### Make sure that everything fits on disk + unless (&Pakfire::check_download_space(@all)) { + exit 1; + } + ### Download first foreach $pak (@all) { &Pakfire::getpak("$pak", ""); @@ -304,6 +316,11 @@ } } + # Make sure that everything fits on disk + unless (&Pakfire::check_download_space(keys %upgradepaks, @deps)) { + exit 1; + } + # Download packages foreach $pak (sort keys %upgradepaks) { &Pakfire::getpak("$pak", ""); @@ -327,6 +344,32 @@ &Pakfire::message("PAKFIRE WARN: No new package upgrades available."); } + # Remove anything that has been left behind in the cache + &Pakfire::tidy_cache(); + + } elsif ("$ARGV[0]" eq "reinstall-core") { + my $tree = &Pakfire::get_tree(); + + unless ($tree eq "testing" || $tree eq "unstable") { + &Pakfire::message("CORE ERROR: Core Updates can only be re-installed on the testing or unstable tree."); + exit 1; + } + + &Pakfire::message("PAKFIRE INFO: Release $Conf::core_mine will be downloaded and re-installed."); + + if ($interactive) { + &Pakfire::message("PAKFIRE INFO: Is this okay? [y/N]"); + my $ret = ; + chomp($ret); + &Pakfire::logger("PAKFIRE INFO: Answer: $ret"); + if ( $ret ne "y" ) { + &Pakfire::message("PAKFIRE ERROR: Re-installation aborted."); + exit 1; + } + } + + &Pakfire::reinstallcore(); + } elsif ("$ARGV[0]" eq "list") { my $count; my $coreupdate = 0; @@ -416,7 +459,7 @@ ### Check if pakfile was actually found if ($metadata{'Installed'} eq "no" && $metadata{'Available'} eq "no") { Pakfire::message("PAKFIRE WARN: Pak '$pak' not found."); - last; + next; } unless (defined $metadata{'Available'}) { @@ -485,7 +528,7 @@ print "Reboot required: $status{'RebootRequired'}\n"; $exitcode += 2 if ($status{'CoreUpdateAvailable'} eq "yes"); - $exitcode += 3 if ($status{'PakUpdatesAvailable'} eq "yes"); + $exitcode += 3 if ($status{'PakUpdatesAvailable'} > 0); $exitcode += 4 if ($status{'RebootRequired'} eq "yes"); exit $exitcode; } else { hooks/post-receive -- IPFire 2.x development tree