IPFire-SCM November 2016

ipfire-scm@lists.ipfire.org

1 participants
20 discussions

[git.ipfire.org] IPFire 2.x development tree branch, next, updated. cd812106b19a146d175fc2e13efcdc68ad04754e
by git＠ipfire.org 29 Nov '16

29 Nov '16

1 0

[git.ipfire.org] IPFire 2.x development tree branch, next, updated. b7f2fe819b1ecf0e0c04e0059b64b67127073d44
by git＠ipfire.org 28 Nov '16

28 Nov '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "IPFire 2.x development tree". The branch, next has been updated via b7f2fe819b1ecf0e0c04e0059b64b67127073d44 (commit) via 0b5b6a594cbe71d0a206176216d0ab1d749ef978 (commit) via 49750f72dee50a2103ead403b16630b67a838231 (commit) via e2b19d984cfa7510edcffc7788ef53cb086cdffb (commit) from 86e9d04bfb73eb256682a567e187fe1e5cdcc3ca (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit b7f2fe819b1ecf0e0c04e0059b64b67127073d44 Author: Michael Tremer <michael.tremer(a)ipfire.org> Date: Mon Nov 28 21:51:13 2016 +0000 core108: Ship updated ddns Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> commit 0b5b6a594cbe71d0a206176216d0ab1d749ef978 Author: Stefan Schantl <stefan.schantl(a)ipfire.org> Date: Fri Oct 28 15:48:22 2016 +0200 ddns: Import patches for schokokeks.org support. Signed-off-by: Stefan Schantl <stefan.schantl(a)ipfire.org> Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> commit 49750f72dee50a2103ead403b16630b67a838231 Author: Michael Tremer <michael.tremer(a)ipfire.org> Date: Mon Nov 28 21:48:21 2016 +0000 Start Core Update 108 Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> commit e2b19d984cfa7510edcffc7788ef53cb086cdffb Author: Michael Tremer <michael.tremer(a)ipfire.org> Date: Mon Nov 28 21:38:29 2016 +0000 strongswan: Update to 5.5.1 Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> ----------------------------------------------------------------------- Summary of changes: config/rootfiles/common/strongswan | 1 + config/rootfiles/{oldcore/106 => core/108}/exclude | 0 .../{oldcore/95 => core/108}/filelists/ddns | 0 config/rootfiles/core/108/filelists/files | 4 ++ .../108}/filelists/i586/strongswan-padlock | 0 .../{oldcore/96 => core/108}/filelists/strongswan | 0 config/rootfiles/core/{107 => 108}/meta | 0 .../rootfiles/{oldcore/99 => core/108}/update.sh | 24 ++++++---- config/rootfiles/{core => oldcore}/107/exclude | 0 .../107/filelists/armv5tel/linux-kirkwood | 0 .../107/filelists/armv5tel/linux-multi | 0 .../107/filelists/armv5tel/linux-rpi | 0 .../{core => oldcore}/107/filelists/files | 0 .../{core => oldcore}/107/filelists/hdparm | 0 .../{core => oldcore}/107/filelists/i586/linux | 0 .../{core => oldcore}/107/filelists/libjpeg | 0 .../{core => oldcore}/107/filelists/libjpeg-compat | 0 .../{core => oldcore}/107/filelists/squid | 0 .../{core => oldcore}/107/filelists/x86_64/linux | 0 config/rootfiles/oldcore/{99 => 107}/meta | 0 config/rootfiles/{core => oldcore}/107/update.sh | 0 lfs/ddns | 3 ++ lfs/strongswan | 4 +- make.sh | 2 +- .../ddns-0001-New-provider-Schokokeks.org.patch | 47 ++++++++++++++++++ ...2-Schokokeks.org-Fix-malformed-update-URL.patch | 55 ++++++++++++++++++++++ 26 files changed, 127 insertions(+), 13 deletions(-) copy config/rootfiles/{oldcore/106 => core/108}/exclude (100%) copy config/rootfiles/{oldcore/95 => core/108}/filelists/ddns (100%) create mode 100644 config/rootfiles/core/108/filelists/files copy config/rootfiles/{oldcore/96 => core/108}/filelists/i586/strongswan-padlock (100%) copy config/rootfiles/{oldcore/96 => core/108}/filelists/strongswan (100%) rename config/rootfiles/core/{107 => 108}/meta (100%) copy config/rootfiles/{oldcore/99 => core/108}/update.sh (89%) rename config/rootfiles/{core => oldcore}/107/exclude (100%) rename config/rootfiles/{core => oldcore}/107/filelists/armv5tel/linux-kirkwood (100%) rename config/rootfiles/{core => oldcore}/107/filelists/armv5tel/linux-multi (100%) rename config/rootfiles/{core => oldcore}/107/filelists/armv5tel/linux-rpi (100%) rename config/rootfiles/{core => oldcore}/107/filelists/files (100%) rename config/rootfiles/{core => oldcore}/107/filelists/hdparm (100%) rename config/rootfiles/{core => oldcore}/107/filelists/i586/linux (100%) rename config/rootfiles/{core => oldcore}/107/filelists/libjpeg (100%) rename config/rootfiles/{core => oldcore}/107/filelists/libjpeg-compat (100%) rename config/rootfiles/{core => oldcore}/107/filelists/squid (100%) rename config/rootfiles/{core => oldcore}/107/filelists/x86_64/linux (100%) copy config/rootfiles/oldcore/{99 => 107}/meta (100%) rename config/rootfiles/{core => oldcore}/107/update.sh (100%) create mode 100644 src/patches/ddns-0001-New-provider-Schokokeks.org.patch create mode 100644 src/patches/ddns-0002-Schokokeks.org-Fix-malformed-update-URL.patch Difference in files: diff --git a/config/rootfiles/common/strongswan b/config/rootfiles/common/strongswan index f81a9c8..38da986 100644 --- a/config/rootfiles/common/strongswan +++ b/config/rootfiles/common/strongswan @@ -72,6 +72,7 @@ etc/swanctl/bliss etc/swanctl/ecdsa etc/swanctl/pkcs12 etc/swanctl/pkcs8 +etc/swanctl/private etc/swanctl/pubkey etc/swanctl/rsa etc/swanctl/swanctl.conf diff --git a/config/rootfiles/core/107/exclude b/config/rootfiles/core/107/exclude deleted file mode 100644 index 1d8d74e..0000000 --- a/config/rootfiles/core/107/exclude +++ /dev/null @@ -1,29 +0,0 @@ -boot/config.txt -boot/grub/grub.cfg -boot/grub/grubenv -etc/alternatives -etc/collectd.custom -etc/default/grub -etc/ipsec.conf -etc/ipsec.secrets -etc/ipsec.user.conf -etc/ipsec.user.secrets -etc/localtime -etc/shadow -etc/snort/snort.conf -etc/ssh/ssh_config -etc/ssh/sshd_config -etc/ssl/openssl.cnf -etc/sudoers -etc/sysconfig/firewall.local -etc/sysconfig/rc.local -etc/udev/rules.d/30-persistent-network.rules -srv/web/ipfire/html/proxy.pac -var/ipfire/dma -var/ipfire/time -var/ipfire/ovpn -var/lib/alternatives -var/lib/unbound/root.key -var/log/cache -var/state/dhcp/dhcpd.leases -var/updatecache diff --git a/config/rootfiles/core/107/filelists/armv5tel/linux-kirkwood b/config/rootfiles/core/107/filelists/armv5tel/linux-kirkwood deleted file mode 120000 index 7217107..0000000 --- a/config/rootfiles/core/107/filelists/armv5tel/linux-kirkwood +++ /dev/null @@ -1 +0,0 @@ -../../../../common/armv5tel/linux-kirkwood \ No newline at end of file diff --git a/config/rootfiles/core/107/filelists/armv5tel/linux-multi b/config/rootfiles/core/107/filelists/armv5tel/linux-multi deleted file mode 120000 index 204eb4c..0000000 --- a/config/rootfiles/core/107/filelists/armv5tel/linux-multi +++ /dev/null @@ -1 +0,0 @@ -../../../../common/armv5tel/linux-multi \ No newline at end of file diff --git a/config/rootfiles/core/107/filelists/armv5tel/linux-rpi b/config/rootfiles/core/107/filelists/armv5tel/linux-rpi deleted file mode 120000 index a651a49..0000000 --- a/config/rootfiles/core/107/filelists/armv5tel/linux-rpi +++ /dev/null @@ -1 +0,0 @@ -../../../../common/armv5tel/linux-rpi \ No newline at end of file diff --git a/config/rootfiles/core/107/filelists/files b/config/rootfiles/core/107/filelists/files deleted file mode 100644 index 94704cf..0000000 --- a/config/rootfiles/core/107/filelists/files +++ /dev/null @@ -1,8 +0,0 @@ -etc/system-release -etc/issue -etc/unbound/unbound.conf -etc/rc.d/init.d/unbound -etc/rc.d/init.d/ntp -srv/web/ipfire/cgi-bin/logs.cgi/log.dat -srv/web/ipfire/cgi-bin/traffic.cgi -var/ipfire/langs diff --git a/config/rootfiles/core/107/filelists/hdparm b/config/rootfiles/core/107/filelists/hdparm deleted file mode 120000 index b644751..0000000 --- a/config/rootfiles/core/107/filelists/hdparm +++ /dev/null @@ -1 +0,0 @@ -../../../common/hdparm \ No newline at end of file diff --git a/config/rootfiles/core/107/filelists/i586/linux b/config/rootfiles/core/107/filelists/i586/linux deleted file mode 120000 index 693ec4b..0000000 --- a/config/rootfiles/core/107/filelists/i586/linux +++ /dev/null @@ -1 +0,0 @@ -../../../../common/i586/linux \ No newline at end of file diff --git a/config/rootfiles/core/107/filelists/libjpeg b/config/rootfiles/core/107/filelists/libjpeg deleted file mode 120000 index 3b1a782..0000000 --- a/config/rootfiles/core/107/filelists/libjpeg +++ /dev/null @@ -1 +0,0 @@ -../../../common/libjpeg \ No newline at end of file diff --git a/config/rootfiles/core/107/filelists/libjpeg-compat b/config/rootfiles/core/107/filelists/libjpeg-compat deleted file mode 120000 index e6ff86d..0000000 --- a/config/rootfiles/core/107/filelists/libjpeg-compat +++ /dev/null @@ -1 +0,0 @@ -../../../common/libjpeg-compat \ No newline at end of file diff --git a/config/rootfiles/core/107/filelists/squid b/config/rootfiles/core/107/filelists/squid deleted file mode 120000 index 2dc8372..0000000 --- a/config/rootfiles/core/107/filelists/squid +++ /dev/null @@ -1 +0,0 @@ -../../../common/squid \ No newline at end of file diff --git a/config/rootfiles/core/107/filelists/x86_64/linux b/config/rootfiles/core/107/filelists/x86_64/linux deleted file mode 120000 index 0615b5b..0000000 --- a/config/rootfiles/core/107/filelists/x86_64/linux +++ /dev/null @@ -1 +0,0 @@ -../../../../common/x86_64/linux \ No newline at end of file diff --git a/config/rootfiles/core/107/meta b/config/rootfiles/core/107/meta deleted file mode 100644 index d547fa8..0000000 --- a/config/rootfiles/core/107/meta +++ /dev/null @@ -1 +0,0 @@ -DEPS="" diff --git a/config/rootfiles/core/107/update.sh b/config/rootfiles/core/107/update.sh deleted file mode 100644 index 276dae5..0000000 --- a/config/rootfiles/core/107/update.sh +++ /dev/null @@ -1,253 +0,0 @@ -#!/bin/bash -############################################################################ -# # -# This file is part of the IPFire Firewall. # -# # -# IPFire is free software; you can redistribute it and/or modify # -# it under the terms of the GNU General Public License as published by # -# the Free Software Foundation; either version 3 of the License, or # -# (at your option) any later version. # -# # -# IPFire is distributed in the hope that it will be useful, # -# but WITHOUT ANY WARRANTY; without even the implied warranty of # -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # -# GNU General Public License for more details. # -# # -# You should have received a copy of the GNU General Public License # -# along with IPFire; if not, write to the Free Software # -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # -# # -# Copyright (C) 2016 IPFire-Team <info(a)ipfire.org>. # -# # -############################################################################ -# -. /opt/pakfire/lib/functions.sh -/usr/local/bin/backupctrl exclude >/dev/null 2>&1 - -function find_device() { - local mountpoint="${1}" - - local root - local dev mp fs flags rest - while read -r dev mp fs flags rest; do - # Skip unwanted entries - [ "${dev}" = "rootfs" ] && continue - - if [ "${mp}" = "${mountpoint}" ] && [ -b "${dev}" ]; then - root="$(basename "${dev}")" - break - fi - done < /proc/mounts - - # Get the actual device from the partition that holds / - while [ -n "${root}" ]; do - if [ -e "/sys/block/${root}" ]; then - echo "${root}" - return 0 - fi - - # Remove last character - root="${root::-1}" - done - - return 1 -} - - -core=107 - -function exit_with_error() { - # Set last succesfull installed core. - echo $(($core-1)) > /opt/pakfire/db/core/mine - /usr/bin/logger -p syslog.emerg -t ipfire \ - "core-update-${core}: $1" - exit $2 -} - -# Remove old core updates from pakfire cache to save space... -for (( i=1; i<=$core; i++ )) -do - rm -f /var/cache/pakfire/core-upgrade-*-$i.ipfire -done - -# -# Do some sanity checks. -case $(uname -r) in - *-ipfire* ) - # Ok. - ;; - * ) - exit_with_error "ERROR cannot update. No IPFire Kernel." 1 - ;; -esac - - -# -# -KVER="xxxKVERxxx" - -# Check diskspace on root -ROOTSPACE=`df / -Pk | sed "s| * | |g" | cut -d" " -f4 | tail -n 1` - -if [ $ROOTSPACE -lt 100000 ]; then - exit_with_error "ERROR cannot update because not enough free space on root." 2 - exit 2 -fi - -echo -echo Update Kernel to $KVER ... -# -# Remove old kernel, configs, initrd, modules, dtb's ... -# -rm -rf /boot/System.map-* -rm -rf /boot/config-* -rm -rf /boot/ipfirerd-* -rm -rf /boot/initramfs-* -rm -rf /boot/vmlinuz-* -rm -rf /boot/uImage-ipfire-* -rm -rf /boot/zImage-ipfire-* -rm -rf /boot/uInit-ipfire-* -rm -rf /boot/dtb-*-ipfire-* -rm -rf /lib/modules - -case "$(uname -m)" in - armv*) - # Backup uEnv.txt if exist - if [ -e /boot/uEnv.txt ]; then - cp -vf /boot/uEnv.txt /boot/uEnv.txt.org - fi - - # work around the u-boot folder detection bug - mkdir -pv /boot/dtb-$KVER-ipfire-kirkwood - mkdir -pv /boot/dtb-$KVER-ipfire-multi - touch /boot/uImage-ipfire-kirkwood - touch /boot/zImage-ipfire-multi - touch /boot/uIinit-ipfire-kirkwood - touch /boot/uIinit-ipfire-multi - ;; -esac - -# Stop services -/etc/init.d/collectd stop -/etc/init.d/snort stop -/etc/init.d/squid stop -/etc/init.d/ipsec stop -/etc/init.d/apache stop - -# Extract files -tar xavf /opt/pakfire/tmp/files* --no-overwrite-dir -p --numeric-owner -C / - -# Remove some old files -rm -f /etc/unbound/interfaces.conf - -# update linker config -ldconfig - -# Check diskspace on boot -BOOTSPACE=`df /boot -Pk | sed "s| * | |g" | cut -d" " -f4 | tail -n 1` - -if [ $BOOTSPACE -lt 1000 ]; then - case $(uname -r) in - *-ipfire-kirkwood ) - # Special handling for old kirkwood images. - # (install only kirkwood kernel) - rm -rf /boot/* - # work around the u-boot folder detection bug - mkdir -pv /boot/dtb-$KVER-ipfire-kirkwood - tar xavf /opt/pakfire/tmp/files* --no-overwrite-dir -p \ - --numeric-owner -C / --wildcards 'boot/*-kirkwood*' - ;; - * ) - /etc/init.d/apache start - exit_with_error "FATAL-ERROR space run out on boot. System is not bootable..." 4 - ;; - esac -fi - -# Update Language cache -/usr/local/bin/update-lang-cache - -# -# Start services -# -/etc/init.d/collectd start -/etc/init.d/apache start -/etc/init.d/squid start -/etc/init.d/snort start -if [ `grep "ENABLED=on" /var/ipfire/vpn/settings` ]; then - /etc/init.d/ipsec start -fi - -# Restart unbound to activate configuration changes -/etc/init.d/unbound restart - -# Delete old QoS enabled indicator -rm -f /var/ipfire/qos/enable - -# Upadate Kernel version uEnv.txt -if [ -e /boot/uEnv.txt ]; then - sed -i -e "s/KVER=.*/KVER=${KVER}/g" /boot/uEnv.txt -fi - -# call user update script (needed for some arm boards) -if [ -e /boot/pakfire-kernel-update ]; then - /boot/pakfire-kernel-update ${KVER} -fi - -case "$(uname -m)" in - i?86) - # Force (re)install pae kernel if pae is supported - rm -rf /opt/pakfire/db/installed/meta-linux-pae - if [ ! "$(grep "^flags.* pae " /proc/cpuinfo)" == "" ]; then - ROOTSPACE=`df / -Pk | sed "s| * | |g" | cut -d" " -f4 | tail -n 1` - BOOTSPACE=`df /boot -Pk | sed "s| * | |g" | cut -d" " -f4 | tail -n 1` - if [ $BOOTSPACE -lt 12000 -o $ROOTSPACE -lt 90000 ]; then - /usr/bin/logger -p syslog.emerg -t ipfire \ - "core-update-${core}: WARNING not enough space for pae kernel." - else - echo "Name: linux-pae" > /opt/pakfire/db/installed/meta-linux-pae - echo "ProgVersion: 0" >> /opt/pakfire/db/installed/meta-linux-pae - echo "Release: 0" >> /opt/pakfire/db/installed/meta-linux-pae - fi - fi - ;; -esac -# -# After pakfire has ended run it again and update the lists and do upgrade -# -echo '#!/bin/bash' > /tmp/pak_update -echo 'while [ "$(ps -A | grep " update.sh")" != "" ]; do' >> /tmp/pak_update -echo ' sleep 1' >> /tmp/pak_update -echo 'done' >> /tmp/pak_update -echo 'while [ "$(ps -A | grep " pakfire")" != "" ]; do' >> /tmp/pak_update -echo ' sleep 1' >> /tmp/pak_update -echo 'done' >> /tmp/pak_update -echo '/opt/pakfire/pakfire update -y --force' >> /tmp/pak_update -echo '/opt/pakfire/pakfire upgrade -y' >> /tmp/pak_update -echo '/opt/pakfire/pakfire upgrade -y' >> /tmp/pak_update -echo '/opt/pakfire/pakfire upgrade -y' >> /tmp/pak_update -echo '/usr/bin/logger -p syslog.emerg -t ipfire "Core-upgrade finished. If you use a customized grub/uboot config"' >> /tmp/pak_update -echo '/usr/bin/logger -p syslog.emerg -t ipfire "Check it before reboot !!!"' >> /tmp/pak_update -echo '/usr/bin/logger -p syslog.emerg -t ipfire " *** Please reboot... *** "' >> /tmp/pak_update -echo 'touch /var/run/need_reboot ' >> /tmp/pak_update -# -killall -KILL pak_update -chmod +x /tmp/pak_update -/tmp/pak_update & - -sync - -# This update need a reboot... -touch /var/run/need_reboot - -# Finish -/etc/init.d/fireinfo start -sendprofile -# Update grub config to display new core version -if [ -e /boot/grub/grub.cfg ]; then - grub-mkconfig -o /boot/grub/grub.cfg -fi -sync - -# Don't report the exitcode last command -exit 0 diff --git a/config/rootfiles/core/108/exclude b/config/rootfiles/core/108/exclude new file mode 100644 index 0000000..7ddeae0 --- /dev/null +++ b/config/rootfiles/core/108/exclude @@ -0,0 +1,28 @@ +boot/config.txt +boot/grub/grub.cfg +boot/grub/grubenv +etc/alternatives +etc/collectd.custom +etc/default/grub +etc/ipsec.conf +etc/ipsec.secrets +etc/ipsec.user.conf +etc/ipsec.user.secrets +etc/localtime +etc/shadow +etc/snort/snort.conf +etc/ssh/ssh_config +etc/ssh/sshd_config +etc/ssl/openssl.cnf +etc/sudoers +etc/sysconfig/firewall.local +etc/sysconfig/rc.local +etc/udev/rules.d/30-persistent-network.rules +srv/web/ipfire/html/proxy.pac +var/ipfire/dma +var/ipfire/time +var/ipfire/ovpn +var/lib/alternatives +var/log/cache +var/state/dhcp/dhcpd.leases +var/updatecache diff --git a/config/rootfiles/core/108/filelists/ddns b/config/rootfiles/core/108/filelists/ddns new file mode 120000 index 0000000..7395164 --- /dev/null +++ b/config/rootfiles/core/108/filelists/ddns @@ -0,0 +1 @@ +../../../common/ddns \ No newline at end of file diff --git a/config/rootfiles/core/108/filelists/files b/config/rootfiles/core/108/filelists/files new file mode 100644 index 0000000..9fc9dbd --- /dev/null +++ b/config/rootfiles/core/108/filelists/files @@ -0,0 +1,4 @@ +etc/system-release +etc/issue +etc/unbound/unbound.conf +srv/web/ipfire/cgi-bin/fwhosts.cgi diff --git a/config/rootfiles/core/108/filelists/i586/strongswan-padlock b/config/rootfiles/core/108/filelists/i586/strongswan-padlock new file mode 120000 index 0000000..2412824 --- /dev/null +++ b/config/rootfiles/core/108/filelists/i586/strongswan-padlock @@ -0,0 +1 @@ +../../../../common/i586/strongswan-padlock \ No newline at end of file diff --git a/config/rootfiles/core/108/filelists/strongswan b/config/rootfiles/core/108/filelists/strongswan new file mode 120000 index 0000000..90c727e --- /dev/null +++ b/config/rootfiles/core/108/filelists/strongswan @@ -0,0 +1 @@ +../../../common/strongswan \ No newline at end of file diff --git a/config/rootfiles/core/108/meta b/config/rootfiles/core/108/meta new file mode 100644 index 0000000..d547fa8 --- /dev/null +++ b/config/rootfiles/core/108/meta @@ -0,0 +1 @@ +DEPS="" diff --git a/config/rootfiles/core/108/update.sh b/config/rootfiles/core/108/update.sh new file mode 100644 index 0000000..045f7e7 --- /dev/null +++ b/config/rootfiles/core/108/update.sh @@ -0,0 +1,66 @@ +#!/bin/bash +############################################################################ +# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 3 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2016 IPFire-Team <info(a)ipfire.org>. # +# # +############################################################################ +# +. /opt/pakfire/lib/functions.sh +/usr/local/bin/backupctrl exclude >/dev/null 2>&1 + +core=108 + +# Remove old core updates from pakfire cache to save space... +for (( i=1; i<=$core; i++ )); do + rm -f /var/cache/pakfire/core-upgrade-*-$i.ipfire +done + +# Stop services +/etc/init.d/ipsec stop + +# Extract files +extract_files + +# update linker config +ldconfig + +# Update Language cache +#/usr/local/bin/update-lang-cache + +# Start services +if grep -q "ENABLED=on" /var/ipfire/vpn/settings; then + /etc/init.d/ipsec start +fi + +# This update need a reboot... +#touch /var/run/need_reboot + +# Finish +/etc/init.d/fireinfo start +sendprofile + +# Update grub config to display new core version +if [ -e /boot/grub/grub.cfg ]; then + grub-mkconfig -o /boot/grub/grub.cfg +fi + +sync + +# Don't report the exitcode last command +exit 0 diff --git a/config/rootfiles/oldcore/107/exclude b/config/rootfiles/oldcore/107/exclude new file mode 100644 index 0000000..1d8d74e --- /dev/null +++ b/config/rootfiles/oldcore/107/exclude @@ -0,0 +1,29 @@ +boot/config.txt +boot/grub/grub.cfg +boot/grub/grubenv +etc/alternatives +etc/collectd.custom +etc/default/grub +etc/ipsec.conf +etc/ipsec.secrets +etc/ipsec.user.conf +etc/ipsec.user.secrets +etc/localtime +etc/shadow +etc/snort/snort.conf +etc/ssh/ssh_config +etc/ssh/sshd_config +etc/ssl/openssl.cnf +etc/sudoers +etc/sysconfig/firewall.local +etc/sysconfig/rc.local +etc/udev/rules.d/30-persistent-network.rules +srv/web/ipfire/html/proxy.pac +var/ipfire/dma +var/ipfire/time +var/ipfire/ovpn +var/lib/alternatives +var/lib/unbound/root.key +var/log/cache +var/state/dhcp/dhcpd.leases +var/updatecache diff --git a/config/rootfiles/oldcore/107/filelists/armv5tel/linux-kirkwood b/config/rootfiles/oldcore/107/filelists/armv5tel/linux-kirkwood new file mode 120000 index 0000000..7217107 --- /dev/null +++ b/config/rootfiles/oldcore/107/filelists/armv5tel/linux-kirkwood @@ -0,0 +1 @@ +../../../../common/armv5tel/linux-kirkwood \ No newline at end of file diff --git a/config/rootfiles/oldcore/107/filelists/armv5tel/linux-multi b/config/rootfiles/oldcore/107/filelists/armv5tel/linux-multi new file mode 120000 index 0000000..204eb4c --- /dev/null +++ b/config/rootfiles/oldcore/107/filelists/armv5tel/linux-multi @@ -0,0 +1 @@ +../../../../common/armv5tel/linux-multi \ No newline at end of file diff --git a/config/rootfiles/oldcore/107/filelists/armv5tel/linux-rpi b/config/rootfiles/oldcore/107/filelists/armv5tel/linux-rpi new file mode 120000 index 0000000..a651a49 --- /dev/null +++ b/config/rootfiles/oldcore/107/filelists/armv5tel/linux-rpi @@ -0,0 +1 @@ +../../../../common/armv5tel/linux-rpi \ No newline at end of file diff --git a/config/rootfiles/oldcore/107/filelists/files b/config/rootfiles/oldcore/107/filelists/files new file mode 100644 index 0000000..94704cf --- /dev/null +++ b/config/rootfiles/oldcore/107/filelists/files @@ -0,0 +1,8 @@ +etc/system-release +etc/issue +etc/unbound/unbound.conf +etc/rc.d/init.d/unbound +etc/rc.d/init.d/ntp +srv/web/ipfire/cgi-bin/logs.cgi/log.dat +srv/web/ipfire/cgi-bin/traffic.cgi +var/ipfire/langs diff --git a/config/rootfiles/oldcore/107/filelists/hdparm b/config/rootfiles/oldcore/107/filelists/hdparm new file mode 120000 index 0000000..b644751 --- /dev/null +++ b/config/rootfiles/oldcore/107/filelists/hdparm @@ -0,0 +1 @@ +../../../common/hdparm \ No newline at end of file diff --git a/config/rootfiles/oldcore/107/filelists/i586/linux b/config/rootfiles/oldcore/107/filelists/i586/linux new file mode 120000 index 0000000..693ec4b --- /dev/null +++ b/config/rootfiles/oldcore/107/filelists/i586/linux @@ -0,0 +1 @@ +../../../../common/i586/linux \ No newline at end of file diff --git a/config/rootfiles/oldcore/107/filelists/libjpeg b/config/rootfiles/oldcore/107/filelists/libjpeg new file mode 120000 index 0000000..3b1a782 --- /dev/null +++ b/config/rootfiles/oldcore/107/filelists/libjpeg @@ -0,0 +1 @@ +../../../common/libjpeg \ No newline at end of file diff --git a/config/rootfiles/oldcore/107/filelists/libjpeg-compat b/config/rootfiles/oldcore/107/filelists/libjpeg-compat new file mode 120000 index 0000000..e6ff86d --- /dev/null +++ b/config/rootfiles/oldcore/107/filelists/libjpeg-compat @@ -0,0 +1 @@ +../../../common/libjpeg-compat \ No newline at end of file diff --git a/config/rootfiles/oldcore/107/filelists/squid b/config/rootfiles/oldcore/107/filelists/squid new file mode 120000 index 0000000..2dc8372 --- /dev/null +++ b/config/rootfiles/oldcore/107/filelists/squid @@ -0,0 +1 @@ +../../../common/squid \ No newline at end of file diff --git a/config/rootfiles/oldcore/107/filelists/x86_64/linux b/config/rootfiles/oldcore/107/filelists/x86_64/linux new file mode 120000 index 0000000..0615b5b --- /dev/null +++ b/config/rootfiles/oldcore/107/filelists/x86_64/linux @@ -0,0 +1 @@ +../../../../common/x86_64/linux \ No newline at end of file diff --git a/config/rootfiles/oldcore/107/meta b/config/rootfiles/oldcore/107/meta new file mode 100644 index 0000000..d547fa8 --- /dev/null +++ b/config/rootfiles/oldcore/107/meta @@ -0,0 +1 @@ +DEPS="" diff --git a/config/rootfiles/oldcore/107/update.sh b/config/rootfiles/oldcore/107/update.sh new file mode 100644 index 0000000..276dae5 --- /dev/null +++ b/config/rootfiles/oldcore/107/update.sh @@ -0,0 +1,253 @@ +#!/bin/bash +############################################################################ +# # +# This file is part of the IPFire Firewall. # +# # +# IPFire is free software; you can redistribute it and/or modify # +# it under the terms of the GNU General Public License as published by # +# the Free Software Foundation; either version 3 of the License, or # +# (at your option) any later version. # +# # +# IPFire is distributed in the hope that it will be useful, # +# but WITHOUT ANY WARRANTY; without even the implied warranty of # +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # +# GNU General Public License for more details. # +# # +# You should have received a copy of the GNU General Public License # +# along with IPFire; if not, write to the Free Software # +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # +# # +# Copyright (C) 2016 IPFire-Team <info(a)ipfire.org>. # +# # +############################################################################ +# +. /opt/pakfire/lib/functions.sh +/usr/local/bin/backupctrl exclude >/dev/null 2>&1 + +function find_device() { + local mountpoint="${1}" + + local root + local dev mp fs flags rest + while read -r dev mp fs flags rest; do + # Skip unwanted entries + [ "${dev}" = "rootfs" ] && continue + + if [ "${mp}" = "${mountpoint}" ] && [ -b "${dev}" ]; then + root="$(basename "${dev}")" + break + fi + done < /proc/mounts + + # Get the actual device from the partition that holds / + while [ -n "${root}" ]; do + if [ -e "/sys/block/${root}" ]; then + echo "${root}" + return 0 + fi + + # Remove last character + root="${root::-1}" + done + + return 1 +} + + +core=107 + +function exit_with_error() { + # Set last succesfull installed core. + echo $(($core-1)) > /opt/pakfire/db/core/mine + /usr/bin/logger -p syslog.emerg -t ipfire \ + "core-update-${core}: $1" + exit $2 +} + +# Remove old core updates from pakfire cache to save space... +for (( i=1; i<=$core; i++ )) +do + rm -f /var/cache/pakfire/core-upgrade-*-$i.ipfire +done + +# +# Do some sanity checks. +case $(uname -r) in + *-ipfire* ) + # Ok. + ;; + * ) + exit_with_error "ERROR cannot update. No IPFire Kernel." 1 + ;; +esac + + +# +# +KVER="xxxKVERxxx" + +# Check diskspace on root +ROOTSPACE=`df / -Pk | sed "s| * | |g" | cut -d" " -f4 | tail -n 1` + +if [ $ROOTSPACE -lt 100000 ]; then + exit_with_error "ERROR cannot update because not enough free space on root." 2 + exit 2 +fi + +echo +echo Update Kernel to $KVER ... +# +# Remove old kernel, configs, initrd, modules, dtb's ... +# +rm -rf /boot/System.map-* +rm -rf /boot/config-* +rm -rf /boot/ipfirerd-* +rm -rf /boot/initramfs-* +rm -rf /boot/vmlinuz-* +rm -rf /boot/uImage-ipfire-* +rm -rf /boot/zImage-ipfire-* +rm -rf /boot/uInit-ipfire-* +rm -rf /boot/dtb-*-ipfire-* +rm -rf /lib/modules + +case "$(uname -m)" in + armv*) + # Backup uEnv.txt if exist + if [ -e /boot/uEnv.txt ]; then + cp -vf /boot/uEnv.txt /boot/uEnv.txt.org + fi + + # work around the u-boot folder detection bug + mkdir -pv /boot/dtb-$KVER-ipfire-kirkwood + mkdir -pv /boot/dtb-$KVER-ipfire-multi + touch /boot/uImage-ipfire-kirkwood + touch /boot/zImage-ipfire-multi + touch /boot/uIinit-ipfire-kirkwood + touch /boot/uIinit-ipfire-multi + ;; +esac + +# Stop services +/etc/init.d/collectd stop +/etc/init.d/snort stop +/etc/init.d/squid stop +/etc/init.d/ipsec stop +/etc/init.d/apache stop + +# Extract files +tar xavf /opt/pakfire/tmp/files* --no-overwrite-dir -p --numeric-owner -C / + +# Remove some old files +rm -f /etc/unbound/interfaces.conf + +# update linker config +ldconfig + +# Check diskspace on boot +BOOTSPACE=`df /boot -Pk | sed "s| * | |g" | cut -d" " -f4 | tail -n 1` + +if [ $BOOTSPACE -lt 1000 ]; then + case $(uname -r) in + *-ipfire-kirkwood ) + # Special handling for old kirkwood images. + # (install only kirkwood kernel) + rm -rf /boot/* + # work around the u-boot folder detection bug + mkdir -pv /boot/dtb-$KVER-ipfire-kirkwood + tar xavf /opt/pakfire/tmp/files* --no-overwrite-dir -p \ + --numeric-owner -C / --wildcards 'boot/*-kirkwood*' + ;; + * ) + /etc/init.d/apache start + exit_with_error "FATAL-ERROR space run out on boot. System is not bootable..." 4 + ;; + esac +fi + +# Update Language cache +/usr/local/bin/update-lang-cache + +# +# Start services +# +/etc/init.d/collectd start +/etc/init.d/apache start +/etc/init.d/squid start +/etc/init.d/snort start +if [ `grep "ENABLED=on" /var/ipfire/vpn/settings` ]; then + /etc/init.d/ipsec start +fi + +# Restart unbound to activate configuration changes +/etc/init.d/unbound restart + +# Delete old QoS enabled indicator +rm -f /var/ipfire/qos/enable + +# Upadate Kernel version uEnv.txt +if [ -e /boot/uEnv.txt ]; then + sed -i -e "s/KVER=.*/KVER=${KVER}/g" /boot/uEnv.txt +fi + +# call user update script (needed for some arm boards) +if [ -e /boot/pakfire-kernel-update ]; then + /boot/pakfire-kernel-update ${KVER} +fi + +case "$(uname -m)" in + i?86) + # Force (re)install pae kernel if pae is supported + rm -rf /opt/pakfire/db/installed/meta-linux-pae + if [ ! "$(grep "^flags.* pae " /proc/cpuinfo)" == "" ]; then + ROOTSPACE=`df / -Pk | sed "s| * | |g" | cut -d" " -f4 | tail -n 1` + BOOTSPACE=`df /boot -Pk | sed "s| * | |g" | cut -d" " -f4 | tail -n 1` + if [ $BOOTSPACE -lt 12000 -o $ROOTSPACE -lt 90000 ]; then + /usr/bin/logger -p syslog.emerg -t ipfire \ + "core-update-${core}: WARNING not enough space for pae kernel." + else + echo "Name: linux-pae" > /opt/pakfire/db/installed/meta-linux-pae + echo "ProgVersion: 0" >> /opt/pakfire/db/installed/meta-linux-pae + echo "Release: 0" >> /opt/pakfire/db/installed/meta-linux-pae + fi + fi + ;; +esac +# +# After pakfire has ended run it again and update the lists and do upgrade +# +echo '#!/bin/bash' > /tmp/pak_update +echo 'while [ "$(ps -A | grep " update.sh")" != "" ]; do' >> /tmp/pak_update +echo ' sleep 1' >> /tmp/pak_update +echo 'done' >> /tmp/pak_update +echo 'while [ "$(ps -A | grep " pakfire")" != "" ]; do' >> /tmp/pak_update +echo ' sleep 1' >> /tmp/pak_update +echo 'done' >> /tmp/pak_update +echo '/opt/pakfire/pakfire update -y --force' >> /tmp/pak_update +echo '/opt/pakfire/pakfire upgrade -y' >> /tmp/pak_update +echo '/opt/pakfire/pakfire upgrade -y' >> /tmp/pak_update +echo '/opt/pakfire/pakfire upgrade -y' >> /tmp/pak_update +echo '/usr/bin/logger -p syslog.emerg -t ipfire "Core-upgrade finished. If you use a customized grub/uboot config"' >> /tmp/pak_update +echo '/usr/bin/logger -p syslog.emerg -t ipfire "Check it before reboot !!!"' >> /tmp/pak_update +echo '/usr/bin/logger -p syslog.emerg -t ipfire " *** Please reboot... *** "' >> /tmp/pak_update +echo 'touch /var/run/need_reboot ' >> /tmp/pak_update +# +killall -KILL pak_update +chmod +x /tmp/pak_update +/tmp/pak_update & + +sync + +# This update need a reboot... +touch /var/run/need_reboot + +# Finish +/etc/init.d/fireinfo start +sendprofile +# Update grub config to display new core version +if [ -e /boot/grub/grub.cfg ]; then + grub-mkconfig -o /boot/grub/grub.cfg +fi +sync + +# Don't report the exitcode last command +exit 0 diff --git a/lfs/ddns b/lfs/ddns index 422f8e3..3d7efa5 100644 --- a/lfs/ddns +++ b/lfs/ddns @@ -71,6 +71,9 @@ $(TARGET) : $(patsubst %,$(DIR_DL)/%,$(objects)) @$(PREBUILD) @rm -rf $(DIR_APP) && cd $(DIR_SRC) && tar axf $(DIR_DL)/$(DL_FILE) + cd $(DIR_APP) && patch -Np1 -i $(DIR_SRC)/src/patches/ddns-0001-New-provider-Schokokeks.org.patch + cd $(DIR_APP) && patch -Np1 -i $(DIR_SRC)/src/patches/ddns-0002-Schokokeks.org-Fix-malformed-update-URL.patch + cd $(DIR_APP) && [ -x "configure" ] || sh ./autogen.sh cd $(DIR_APP) && ./configure \ --prefix=/usr \ diff --git a/lfs/strongswan b/lfs/strongswan index 17c1a01..9e8f155 100644 --- a/lfs/strongswan +++ b/lfs/strongswan @@ -24,7 +24,7 @@ include Config -VER = 5.5.0 +VER = 5.5.1 THISAPP = strongswan-$(VER) DL_FILE = $(THISAPP).tar.bz2 @@ -48,7 +48,7 @@ objects = $(DL_FILE) $(DL_FILE) = $(DL_FROM)/$(DL_FILE) -$(DL_FILE)_MD5 = a96fa7eb6c62b40143dadb064b6bd586 +$(DL_FILE)_MD5 = 4eba9474f7dc6c8c8d7037261358e68d install : $(TARGET) diff --git a/make.sh b/make.sh index 4b7beb8..4c5996a 100755 --- a/make.sh +++ b/make.sh @@ -25,7 +25,7 @@ NAME="IPFire" # Software name SNAME="ipfire" # Short name VERSION="2.19" # Version number -CORE="107" # Core Level (Filename) +CORE="108" # Core Level (Filename) PAKFIRE_CORE="107" # Core Level (PAKFIRE) GIT_BRANCH=`git rev-parse --abbrev-ref HEAD` # Git Branch SLOGAN="www.ipfire.org" # Software slogan diff --git a/src/patches/ddns-0001-New-provider-Schokokeks.org.patch b/src/patches/ddns-0001-New-provider-Schokokeks.org.patch new file mode 100644 index 0000000..be123a5 --- /dev/null +++ b/src/patches/ddns-0001-New-provider-Schokokeks.org.patch @@ -0,0 +1,47 @@ +From 521c9d90f4e879ef3d9e1590f29e27990011ae46 Mon Sep 17 00:00:00 2001 +From: Steffen Peters <sauron99(a)gmx.de> +Date: Mon, 4 Jul 2016 22:14:10 +0200 +Subject: [PATCH 185/185] New provider: Schokokeks.org + +Signed-off-by: Steffen Peters <sauron99(a)gmx.de> +Signed-off-by: Stefan Schantl <stefan.schantl(a)ipfire.org> +--- + README | 1 + + src/ddns/providers.py | 12 ++++++++++++ + 2 files changed, 13 insertions(+) + +diff --git a/README b/README +index d8027a4..cedbf21 100644 +--- a/README ++++ b/README +@@ -75,6 +75,7 @@ SUPPORTED PROVIDERS: + opendns.com + ovh.com + regfish.com ++ schokokeks.org + selfhost.de + spdns.org + strato.com +diff --git a/src/ddns/providers.py b/src/ddns/providers.py +index 6b25cb6..c482dad 100644 +--- a/src/ddns/providers.py ++++ b/src/ddns/providers.py +@@ -1687,3 +1687,15 @@ class DDNSProviderZZZZ(DDNSProvider): + + # If we got here, some other update error happened. + raise DDNSUpdateError ++ ++class DDNSProviderSchokokeksDNS(DDNSProtocolDynDNS2, DDNSProvider): ++ handle = "schokokeks.org" ++ name = "Schokokeks" ++ website = "http://www.schokokeks.org/" ++ protocols = ("ipv4",) ++ ++ # Information about the format of the request is to be found ++ # https://wiki.schokokeks.org/DynDNS ++ ++ url = "https://dyndns.schokokeks.org/nic/update?myip=<ipaddr>" ++ +-- +2.7.4 + diff --git a/src/patches/ddns-0002-Schokokeks.org-Fix-malformed-update-URL.patch b/src/patches/ddns-0002-Schokokeks.org-Fix-malformed-update-URL.patch new file mode 100644 index 0000000..e00dcf6 --- /dev/null +++ b/src/patches/ddns-0002-Schokokeks.org-Fix-malformed-update-URL.patch @@ -0,0 +1,55 @@ +From f77e6bc92825d65e881d5dc7fc443139278c0d5f Mon Sep 17 00:00:00 2001 +From: Stefan Schantl <stefan.schantl(a)ipfire.org> +Date: Fri, 28 Oct 2016 12:35:20 +0200 +Subject: [PATCH 3/3] Schockokeks.org: Fix malformed update URL. + +* Move Provider Class into correct alphabetical order. + +Signed-off-by: Stefan Schantl <stefan.schantl(a)ipfire.org> +Reviewed-by: Michael Tremer <michael.tremer(a)ipfire.org> +Signed-off-by: Stefan Schantl <stefan.schantl(a)ipfire.org> +--- + src/ddns/providers.py | 23 +++++++++++------------ + 1 file changed, 11 insertions(+), 12 deletions(-) + +diff --git a/src/ddns/providers.py b/src/ddns/providers.py +index c482dad..2c30d42 100644 +--- a/src/ddns/providers.py ++++ b/src/ddns/providers.py +@@ -1424,6 +1424,17 @@ class DDNSProviderRegfish(DDNSProvider): + raise DDNSUpdateError + + ++class DDNSProviderSchokokeksDNS(DDNSProtocolDynDNS2, DDNSProvider): ++ handle = "schokokeks.org" ++ name = "Schokokeks" ++ website = "http://www.schokokeks.org/" ++ protocols = ("ipv4",) ++ ++ # Information about the format of the request is to be found ++ # https://wiki.schokokeks.org/DynDNS ++ url = "https://dyndns.schokokeks.org/nic/update" ++ ++ + class DDNSProviderSelfhost(DDNSProtocolDynDNS2, DDNSProvider): + handle = "selfhost.de" + name = "Selfhost.de" +@@ -1687,15 +1698,3 @@ class DDNSProviderZZZZ(DDNSProvider): + + # If we got here, some other update error happened. + raise DDNSUpdateError +- +-class DDNSProviderSchokokeksDNS(DDNSProtocolDynDNS2, DDNSProvider): +- handle = "schokokeks.org" +- name = "Schokokeks" +- website = "http://www.schokokeks.org/" +- protocols = ("ipv4",) +- +- # Information about the format of the request is to be found +- # https://wiki.schokokeks.org/DynDNS +- +- url = "https://dyndns.schokokeks.org/nic/update?myip=<ipaddr>" +- +-- +2.7.4 + hooks/post-receive -- IPFire 2.x development tree

1 0

[git.ipfire.org] IPFire 3.x development tree branch, master, updated. 449502be43fa31e974cf547c65a2cc2b8a588dce
by git＠ipfire.org 28 Nov '16

28 Nov '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "IPFire 3.x development tree". The branch, master has been updated via 449502be43fa31e974cf547c65a2cc2b8a588dce (commit) via 31665633109eeb20192ab37c76c3002663ded6d4 (commit) from fb50fbb2b4e08194db20f0f7bfd0ce415786b18a (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 449502be43fa31e974cf547c65a2cc2b8a588dce Author: Stefan Schantl <stefan.schantl(a)ipfire.org> Date: Mon Nov 28 13:20:15 2016 +0100 boost: Fix RPATH issue on i686. Fixes #11176. Signed-off-by: Stefan Schantl <stefan.schantl(a)ipfire.org> Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> commit 31665633109eeb20192ab37c76c3002663ded6d4 Author: Michael Tremer <michael.tremer(a)ipfire.org> Date: Sat Nov 26 20:45:06 2016 +0000 python-urlgrabber: Update to 3.10.1 Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> ----------------------------------------------------------------------- Summary of changes: boost/boost.nm | 2 +- boost/patches/boost-1.60.0-no-rpath.patch0 | 21 + python-urlgrabber/patches/urlgrabber-HEAD.patch | 2343 ----------------------- python-urlgrabber/python-urlgrabber.nm | 6 +- 4 files changed, 25 insertions(+), 2347 deletions(-) create mode 100644 boost/patches/boost-1.60.0-no-rpath.patch0 delete mode 100644 python-urlgrabber/patches/urlgrabber-HEAD.patch Difference in files: diff --git a/boost/boost.nm b/boost/boost.nm index 0a6af33..1e6038c 100644 --- a/boost/boost.nm +++ b/boost/boost.nm @@ -8,7 +8,7 @@ ver_major = 1 ver_minor = 60 ver_plevel = 0 version = %{ver_major}.%{ver_minor}.%{ver_plevel} -release = 1 +release = 2 groups = System/Libraries url = http://www.boost.org/ diff --git a/boost/patches/boost-1.60.0-no-rpath.patch0 b/boost/patches/boost-1.60.0-no-rpath.patch0 new file mode 100644 index 0000000..eed65c2 --- /dev/null +++ b/boost/patches/boost-1.60.0-no-rpath.patch0 @@ -0,0 +1,21 @@ +diff -up tools/build/src/tools/gcc.jam.rpath tools/build/src/tools/gcc.jam +--- tools/build/src/tools/gcc.jam.rpath 2016-05-27 13:30:01.092192721 -0500 ++++ tools/build/src/tools/gcc.jam 2016-05-27 13:30:46.686987585 -0500 +@@ -952,7 +952,7 @@ rule link ( targets * : sources * : prop + + actions link bind LIBRARIES + { +- "$(CONFIG_COMMAND)" -L"$(LINKPATH)" -Wl,$(RPATH_OPTION:E=-R)$(SPACE)-Wl,$(RPATH) -Wl,-rpath-link$(SPACE)-Wl,"$(RPATH_LINK)" -o "$(<)" $(START-GROUP) "$(>)" "$(LIBRARIES)" $(FINDLIBS-ST-PFX) -l$(FINDLIBS-ST) $(FINDLIBS-SA-PFX) -l$(FINDLIBS-SA) $(END-GROUP) $(OPTIONS) $(USER_OPTIONS) ++ "$(CONFIG_COMMAND)" -L"$(LINKPATH)" -o "$(<)" $(START-GROUP) "$(>)" "$(LIBRARIES)" $(FINDLIBS-ST-PFX) -l$(FINDLIBS-ST) $(FINDLIBS-SA-PFX) -l$(FINDLIBS-SA) $(END-GROUP) $(OPTIONS) $(USER_OPTIONS) + } + + +@@ -1018,7 +1018,7 @@ rule link.dll ( targets * : sources * : + # Differs from 'link' above only by -shared. + actions link.dll bind LIBRARIES + { +- "$(CONFIG_COMMAND)" -L"$(LINKPATH)" -Wl,$(RPATH_OPTION:E=-R)$(SPACE)-Wl,$(RPATH) "$(.IMPLIB-COMMAND)$(<[1])" -o "$(<[-1])" $(HAVE_SONAME)-Wl,$(SONAME_OPTION)$(SPACE)-Wl,$(<[-1]:D=) -shared $(START-GROUP) "$(>)" "$(LIBRARIES)" $(FINDLIBS-ST-PFX) -l$(FINDLIBS-ST) $(FINDLIBS-SA-PFX) -l$(FINDLIBS-SA) $(END-GROUP) $(OPTIONS) $(USER_OPTIONS) ++ "$(CONFIG_COMMAND)" -L"$(LINKPATH)" "$(.IMPLIB-COMMAND)$(<[1])" -o "$(<[-1])" $(HAVE_SONAME)-Wl,$(SONAME_OPTION)$(SPACE)-Wl,$(<[-1]:D=) -shared $(START-GROUP) "$(>)" "$(LIBRARIES)" $(FINDLIBS-ST-PFX) -l$(FINDLIBS-ST) $(FINDLIBS-SA-PFX) -l$(FINDLIBS-SA) $(END-GROUP) $(OPTIONS) $(USER_OPTIONS) + } + + rule setup-threading ( targets * : sources * : properties * ) diff --git a/python-urlgrabber/patches/urlgrabber-HEAD.patch b/python-urlgrabber/patches/urlgrabber-HEAD.patch deleted file mode 100644 index aaf9cbc..0000000 --- a/python-urlgrabber/patches/urlgrabber-HEAD.patch +++ /dev/null @@ -1,2343 +0,0 @@ -diff --git a/.gitignore b/.gitignore -new file mode 100644 -index 0000000..1ffe416 ---- /dev/null -+++ b/.gitignore -@@ -0,0 +1,7 @@ -+*.py[co] -+MANIFEST -+dist -+build -+*.kdev* -+*.kateproject -+ipython.log* -diff --git a/scripts/urlgrabber b/scripts/urlgrabber -index 518e512..09cd896 100644 ---- a/scripts/urlgrabber -+++ b/scripts/urlgrabber -@@ -115,6 +115,7 @@ options: - including quotes in the case of strings. - e.g. --user_agent='"foobar/2.0"' - -+ --output FILE - -o FILE write output to FILE, otherwise the basename of the - url will be used - -O print the names of saved files to STDOUT -@@ -170,12 +171,17 @@ class client_options: - return ug_options, ug_defaults - - def process_command_line(self): -- short_options = 'vd:hoOpD' -+ short_options = 'vd:ho:OpD' - long_options = ['profile', 'repeat=', 'verbose=', -- 'debug=', 'help', 'progress'] -+ 'debug=', 'help', 'progress', 'output='] - ug_long = [ o + '=' for o in self.ug_options ] -- optlist, args = getopt.getopt(sys.argv[1:], short_options, -- long_options + ug_long) -+ try: -+ optlist, args = getopt.getopt(sys.argv[1:], short_options, -+ long_options + ug_long) -+ except getopt.GetoptError, e: -+ print >>sys.stderr, "Error:", e -+ self.help([], ret=1) -+ - self.verbose = 0 - self.debug = None - self.outputfile = None -@@ -193,6 +199,7 @@ class client_options: - if o == '--verbose': self.verbose = v - if o == '-v': self.verbose += 1 - if o == '-o': self.outputfile = v -+ if o == '--output': self.outputfile = v - if o == '-p' or o == '--progress': self.progress = 1 - if o == '-d' or o == '--debug': self.debug = v - if o == '--profile': self.profile = 1 -@@ -222,7 +229,7 @@ class client_options: - print "ERROR: cannot use -o when grabbing multiple files" - sys.exit(1) - -- def help(self, args): -+ def help(self, args, ret=0): - if not args: - print MAINHELP - else: -@@ -234,7 +241,7 @@ class client_options: - self.help_ug_option(a) - else: - print 'ERROR: no help on command "%s"' % a -- sys.exit(0) -+ sys.exit(ret) - - def help_doc(self): - print __doc__ -diff --git a/scripts/urlgrabber-ext-down b/scripts/urlgrabber-ext-down -new file mode 100755 -index 0000000..3dafb12 ---- /dev/null -+++ b/scripts/urlgrabber-ext-down -@@ -0,0 +1,75 @@ -+#! /usr/bin/python -+# A very simple external downloader -+# Copyright 2011-2012 Zdenek Pavlas -+ -+# This library is free software; you can redistribute it and/or -+# modify it under the terms of the GNU Lesser General Public -+# License as published by the Free Software Foundation; either -+# version 2.1 of the License, or (at your option) any later version. -+# -+# This library is distributed in the hope that it will be useful, -+# but WITHOUT ANY WARRANTY; without even the implied warranty of -+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -+# Lesser General Public License for more details. -+# -+# You should have received a copy of the GNU Lesser General Public -+# License along with this library; if not, write to the -+# Free Software Foundation, Inc., -+# 59 Temple Place, Suite 330, -+# Boston, MA 02111-1307 USA -+ -+import time, os, errno, sys -+from urlgrabber.grabber import \ -+ _readlines, URLGrabberOptions, _loads, \ -+ PyCurlFileObject, URLGrabError -+ -+def write(fmt, *arg): -+ try: os.write(1, fmt % arg) -+ except OSError, e: -+ if e.args[0] != errno.EPIPE: raise -+ sys.exit(1) -+ -+class ProxyProgress: -+ def start(self, *d1, **d2): -+ self.next_update = 0 -+ def update(self, _amount_read): -+ t = time.time() -+ if t < self.next_update: return -+ self.next_update = t + 0.31 -+ write('%d %d\n', self._id, _amount_read) -+ -+def main(): -+ import signal -+ signal.signal(signal.SIGINT, lambda n, f: sys.exit(1)) -+ cnt = 0 -+ while True: -+ lines = _readlines(0) -+ if not lines: break -+ for line in lines: -+ cnt += 1 -+ opts = URLGrabberOptions() -+ opts._id = cnt -+ for k in line.split(' '): -+ k, v = k.split('=', 1) -+ setattr(opts, k, _loads(v)) -+ if opts.progress_obj: -+ opts.progress_obj = ProxyProgress() -+ opts.progress_obj._id = cnt -+ -+ dlsz = dltm = 0 -+ try: -+ fo = PyCurlFileObject(opts.url, opts.filename, opts) -+ fo._do_grab() -+ fo.fo.close() -+ size = fo._amount_read -+ if fo._tm_last: -+ dlsz = fo._tm_last[0] - fo._tm_first[0] -+ dltm = fo._tm_last[1] - fo._tm_first[1] -+ ug_err = 'OK' -+ except URLGrabError, e: -+ size = 0 -+ ug_err = '%d %s' % e.args -+ write('%d %d %d %.3f %s\n', opts._id, size, dlsz, dltm, ug_err) -+ -+if __name__ == '__main__': -+ main() -diff --git a/setup.py b/setup.py -index d0b87b8..bfa4a18 100644 ---- a/setup.py -+++ b/setup.py -@@ -15,8 +15,10 @@ url = _urlgrabber.__url__ - packages = ['urlgrabber'] - package_dir = {'urlgrabber':'urlgrabber'} - scripts = ['scripts/urlgrabber'] --data_files = [('share/doc/' + name + '-' + version, -- ['README','LICENSE', 'TODO', 'ChangeLog'])] -+data_files = [ -+ ('share/doc/' + name + '-' + version, ['README','LICENSE', 'TODO', 'ChangeLog']), -+ ('libexec', ['scripts/urlgrabber-ext-down']), -+] - options = { 'clean' : { 'all' : 1 } } - classifiers = [ - 'Development Status :: 4 - Beta', -diff --git a/test/base_test_code.py b/test/base_test_code.py -index 50c6348..5fb43f9 100644 ---- a/test/base_test_code.py -+++ b/test/base_test_code.py -@@ -1,6 +1,6 @@ - from munittest import * - --base_http = 'http://www.linux.duke.edu/projects/urlgrabber/test/' -+base_http = 'http://urlgrabber.baseurl.org/test/' - base_ftp = 'ftp://localhost/test/' - - # set to a proftp server only. we're working around a couple of -diff --git a/urlgrabber/byterange.py b/urlgrabber/byterange.py -index 3e5f3b7..8eeaeda 100644 ---- a/urlgrabber/byterange.py -+++ b/urlgrabber/byterange.py -@@ -68,7 +68,7 @@ class HTTPRangeHandler(urllib2.BaseHandler): - - def http_error_416(self, req, fp, code, msg, hdrs): - # HTTP's Range Not Satisfiable error -- raise RangeError('Requested Range Not Satisfiable') -+ raise RangeError(9, 'Requested Range Not Satisfiable') - - class HTTPSRangeHandler(HTTPRangeHandler): - """ Range Header support for HTTPS. """ -@@ -208,7 +208,7 @@ class RangeableFileObject: - bufsize = offset - pos - buf = self.fo.read(bufsize) - if len(buf) != bufsize: -- raise RangeError('Requested Range Not Satisfiable') -+ raise RangeError(9, 'Requested Range Not Satisfiable') - pos+= bufsize - - class FileRangeHandler(urllib2.FileHandler): -@@ -238,7 +238,7 @@ class FileRangeHandler(urllib2.FileHandler): - (fb,lb) = brange - if lb == '': lb = size - if fb < 0 or fb > size or lb > size: -- raise RangeError('Requested Range Not Satisfiable') -+ raise RangeError(9, 'Requested Range Not Satisfiable') - size = (lb - fb) - fo = RangeableFileObject(fo, (fb,lb)) - headers = mimetools.Message(StringIO( -@@ -318,12 +318,12 @@ class FTPRangeHandler(urllib2.FTPHandler): - (fb,lb) = range_tup - if lb == '': - if retrlen is None or retrlen == 0: -- raise RangeError('Requested Range Not Satisfiable due to unobtainable file length.') -+ raise RangeError(9, 'Requested Range Not Satisfiable due to unobtainable file length.') - lb = retrlen - retrlen = lb - fb - if retrlen < 0: - # beginning of range is larger than file -- raise RangeError('Requested Range Not Satisfiable') -+ raise RangeError(9, 'Requested Range Not Satisfiable') - else: - retrlen = lb - fb - fp = RangeableFileObject(fp, (0,retrlen)) -@@ -458,6 +458,6 @@ def range_tuple_normalize(range_tup): - # check if range is over the entire file - if (fb,lb) == (0,''): return None - # check that the range is valid -- if lb < fb: raise RangeError('Invalid byte range: %s-%s' % (fb,lb)) -+ if lb < fb: raise RangeError(9, 'Invalid byte range: %s-%s' % (fb,lb)) - return (fb,lb) - -diff --git a/urlgrabber/grabber.py b/urlgrabber/grabber.py -index e090e90..6ce9861 100644 ---- a/urlgrabber/grabber.py -+++ b/urlgrabber/grabber.py -@@ -49,11 +49,26 @@ GENERAL ARGUMENTS (kwargs) - progress_obj = None - - a class instance that supports the following methods: -- po.start(filename, url, basename, length, text) -+ po.start(filename, url, basename, size, now, text) - # length will be None if unknown - po.update(read) # read == bytes read so far - po.end() - -+ multi_progress_obj = None -+ -+ a class instance that supports the following methods: -+ mo.start(total_files, total_size) -+ mo.newMeter() => meter -+ mo.removeMeter(meter) -+ mo.end() -+ -+ The 'meter' object is similar to progress_obj, but multiple -+ instances may be created and updated at the same time. -+ -+ When downloading multiple files in parallel and multi_progress_obj -+ is None progress_obj is used in compatibility mode: finished files -+ are shown but there's no in-progress display. -+ - text = None - - specifies alternative text to be passed to the progress meter -@@ -68,14 +83,14 @@ GENERAL ARGUMENTS (kwargs) - (which can be set on default_grabber.throttle) is used. See - BANDWIDTH THROTTLING for more information. - -- timeout = None -+ timeout = 300 - -- a positive float expressing the number of seconds to wait for socket -- operations. If the value is None or 0.0, socket operations will block -- forever. Setting this option causes urlgrabber to call the settimeout -- method on the Socket object used for the request. See the Python -- documentation on settimeout for more information. -- http://www.python.org/doc/current/lib/socket-objects.html -+ a positive integer expressing the number of seconds to wait before -+ timing out attempts to connect to a server. If the value is None -+ or 0, connection attempts will not time out. The timeout is passed -+ to the underlying pycurl object as its CONNECTTIMEOUT option, see -+ the curl documentation on CURLOPT_CONNECTTIMEOUT for more information. -+ http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTCONNECTTIMEOUT - - bandwidth = 0 - -@@ -143,8 +158,12 @@ GENERAL ARGUMENTS (kwargs) - note that proxy authentication information may be provided using - normal URL constructs: - proxies={ 'http' : 'http://user:host@foo:3128' } -- Lastly, if proxies is None, the default environment settings will -- be used. -+ -+ libproxy = False -+ -+ Use the libproxy module (if installed) to find proxies. -+ The libproxy code is only used if the proxies dictionary -+ does not provide any proxies. - - prefix = None - -@@ -198,6 +217,12 @@ GENERAL ARGUMENTS (kwargs) - control, you should probably subclass URLParser and pass it in via - the 'urlparser' option. - -+ username = None -+ username to use for simple http auth - is automatically quoted for special characters -+ -+ password = None -+ password to use for simple http auth - is automatically quoted for special characters -+ - ssl_ca_cert = None - - this option can be used if M2Crypto is available and will be -@@ -211,43 +236,75 @@ GENERAL ARGUMENTS (kwargs) - No-op when using the curl backend (default) - - -- self.ssl_verify_peer = True -+ ssl_verify_peer = True - - Check the server's certificate to make sure it is valid with what our CA validates - -- self.ssl_verify_host = True -+ ssl_verify_host = True - - Check the server's hostname to make sure it matches the certificate DN - -- self.ssl_key = None -+ ssl_key = None - - Path to the key the client should use to connect/authenticate with - -- self.ssl_key_type = 'PEM' -+ ssl_key_type = 'PEM' - - PEM or DER - format of key - -- self.ssl_cert = None -+ ssl_cert = None - - Path to the ssl certificate the client should use to to authenticate with - -- self.ssl_cert_type = 'PEM' -+ ssl_cert_type = 'PEM' - - PEM or DER - format of certificate - -- self.ssl_key_pass = None -+ ssl_key_pass = None - - password to access the ssl_key - -- self.size = None -+ size = None - - size (in bytes) or Maximum size of the thing being downloaded. - This is mostly to keep us from exploding with an endless datastream - -- self.max_header_size = 2097152 -+ max_header_size = 2097152 - - Maximum size (in bytes) of the headers. - -+ ip_resolve = 'whatever' -+ -+ What type of name to IP resolving to use, default is to do both IPV4 and -+ IPV6. -+ -+ async = (key, limit) -+ -+ When this option is set, the urlgrab() is not processed immediately -+ but queued. parallel_wait() then processes grabs in parallel, limiting -+ the numer of connections in each 'key' group to at most 'limit'. -+ -+ max_connections -+ -+ The global connection limit. -+ -+ timedhosts -+ -+ The filename of the host download statistics. If defined, urlgrabber -+ will update the stats at the end of every download. At the end of -+ parallel_wait(), the updated stats are saved. If synchronous grabs -+ are used, you should call th_save(). -+ -+ default_speed, half_life -+ -+ These options only affect the async mirror selection code. -+ The default_speed option sets the speed estimate for mirrors -+ we have never downloaded from, and defaults to 1 MBps. -+ -+ The speed estimate also drifts exponentially from the speed -+ actually measured to the default speed, with default -+ period of 30 days. -+ - - RETRY RELATED ARGUMENTS - -@@ -328,6 +385,15 @@ RETRY RELATED ARGUMENTS - but it cannot (without severe trickiness) prevent the exception - from being raised. - -+ failfunc = None -+ -+ The callback that gets called when urlgrab request fails. -+ If defined, urlgrab() calls it instead of raising URLGrabError. -+ Callback syntax is identical to failure_callback. -+ -+ Contrary to failure_callback, it's called only once. It's primary -+ purpose is to use urlgrab() without a try/except block. -+ - interrupt_callback = None - - This callback is called if KeyboardInterrupt is received at any -@@ -420,6 +486,7 @@ import time - import string - import urllib - import urllib2 -+from httplib import responses - import mimetools - import thread - import types -@@ -428,9 +495,17 @@ import pycurl - from ftplib import parse150 - from StringIO import StringIO - from httplib import HTTPException --import socket -+import socket, select, fcntl - from byterange import range_tuple_normalize, range_tuple_to_header, RangeError - -+try: -+ import xattr -+ if not hasattr(xattr, 'set'): -+ xattr = None # This is a "newer" API. -+except ImportError: -+ xattr = None -+ -+ - ######################################################################## - # MODULE INITIALIZATION - ######################################################################## -@@ -439,6 +514,12 @@ try: - except: - __version__ = '???' - -+try: -+ # this part isn't going to do much - need to talk to gettext -+ from i18n import _ -+except ImportError, msg: -+ def _(st): return st -+ - ######################################################################## - # functions for debugging output. These functions are here because they - # are also part of the module initialization. -@@ -504,6 +585,7 @@ def _init_default_logger(logspec=None): - else: handler = logging.FileHandler(filename) - handler.setFormatter(formatter) - DBOBJ = logging.getLogger('urlgrabber') -+ DBOBJ.propagate = False - DBOBJ.addHandler(handler) - DBOBJ.setLevel(level) - except (KeyError, ImportError, ValueError): -@@ -512,8 +594,8 @@ def _init_default_logger(logspec=None): - - def _log_package_state(): - if not DEBUG: return -- DEBUG.info('urlgrabber version = %s' % __version__) -- DEBUG.info('trans function "_" = %s' % _) -+ DEBUG.debug('urlgrabber version = %s' % __version__) -+ DEBUG.debug('trans function "_" = %s' % _) - - _init_default_logger() - _log_package_state() -@@ -527,6 +609,29 @@ def _(st): - # END MODULE INITIALIZATION - ######################################################################## - -+######################################################################## -+# UTILITY FUNCTIONS -+######################################################################## -+ -+# These functions are meant to be utilities for the urlgrabber library to use. -+ -+def _to_utf8(obj, errors='replace'): -+ '''convert 'unicode' to an encoded utf-8 byte string ''' -+ # stolen from yum.i18n -+ if isinstance(obj, unicode): -+ obj = obj.encode('utf-8', errors) -+ return obj -+ -+def exception2msg(e): -+ try: -+ return str(e) -+ except UnicodeEncodeError: -+ # always use byte strings -+ return unicode(e).encode('utf8') -+ -+######################################################################## -+# END UTILITY FUNCTIONS -+######################################################################## - - - class URLGrabError(IOError): -@@ -662,6 +767,7 @@ class URLParser: - opts.quote = 0 --> do not quote it - opts.quote = None --> guess - """ -+ url = _to_utf8(url) - quote = opts.quote - - if opts.prefix: -@@ -768,6 +874,41 @@ class URLGrabberOptions: - else: # throttle is a float - return self.bandwidth * self.throttle - -+ def find_proxy(self, url, scheme): -+ """Find the proxy to use for this URL. -+ Use the proxies dictionary first, then libproxy. -+ """ -+ self.proxy = None -+ if scheme not in ('ftp', 'http', 'https'): -+ return -+ -+ if self.proxies: -+ proxy = self.proxies.get(scheme) -+ if proxy is None: -+ if scheme == 'http': -+ proxy = self.proxies.get('https') -+ elif scheme == 'https': -+ proxy = self.proxies.get('http') -+ if proxy == '_none_': -+ proxy = '' -+ self.proxy = proxy -+ return -+ -+ if self.libproxy: -+ global _libproxy_cache -+ if _libproxy_cache is None: -+ try: -+ import libproxy -+ _libproxy_cache = libproxy.ProxyFactory() -+ except: -+ _libproxy_cache = False -+ if _libproxy_cache: -+ for proxy in _libproxy_cache.getProxies(url): -+ if proxy.startswith('http://'): -+ if DEBUG: DEBUG.info('using proxy "%s" for url %s' % (proxy, url)) -+ self.proxy = proxy -+ break -+ - def derive(self, **kwargs): - """Create a derived URLGrabberOptions instance. - This method creates a new instance and overrides the -@@ -791,30 +932,37 @@ class URLGrabberOptions: - provided here. - """ - self.progress_obj = None -+ self.multi_progress_obj = None - self.throttle = 1.0 - self.bandwidth = 0 - self.retry = None - self.retrycodes = [-1,2,4,5,6,7] - self.checkfunc = None -+ self.failfunc = _do_raise - self.copy_local = 0 - self.close_connection = 0 - self.range = None - self.user_agent = 'urlgrabber/%s' % __version__ -+ self.ip_resolve = None - self.keepalive = 1 - self.proxies = None -+ self.libproxy = False -+ self.proxy = None - self.reget = None - self.failure_callback = None - self.interrupt_callback = None - self.prefix = None - self.opener = None - self.cache_openers = True -- self.timeout = None -+ self.timeout = 300 - self.text = None - self.http_headers = None - self.ftp_headers = None - self.data = None - self.urlparser = URLParser() - self.quote = None -+ self.username = None -+ self.password = None - self.ssl_ca_cert = None # sets SSL_CAINFO - path to certdb - self.ssl_context = None # no-op in pycurl - self.ssl_verify_peer = True # check peer's cert for authenticityb -@@ -827,6 +975,12 @@ class URLGrabberOptions: - self.size = None # if we know how big the thing we're getting is going - # to be. this is ultimately a MAXIMUM size for the file - self.max_header_size = 2097152 #2mb seems reasonable for maximum header size -+ self.async = None # blocking by default -+ self.mirror_group = None -+ self.max_connections = 5 -+ self.timedhosts = None -+ self.half_life = 30*24*60*60 # 30 days -+ self.default_speed = 1e6 # 1 MBit - - def __repr__(self): - return self.format() -@@ -846,7 +1000,18 @@ class URLGrabberOptions: - s = s + indent + '}' - return s - --class URLGrabber: -+def _do_raise(obj): -+ raise obj.exception -+ -+def _run_callback(cb, obj): -+ if not cb: -+ return -+ if callable(cb): -+ return cb(obj) -+ cb, arg, karg = cb -+ return cb(obj, *arg, **karg) -+ -+class URLGrabber(object): - """Provides easy opening of URLs with a variety of options. - - All options are specified as kwargs. Options may be specified when -@@ -872,7 +1037,6 @@ class URLGrabber: - # beware of infinite loops :) - tries = tries + 1 - exception = None -- retrycode = None - callback = None - if DEBUG: DEBUG.info('attempt %i/%s: %s', - tries, opts.retry, args[0]) -@@ -883,54 +1047,62 @@ class URLGrabber: - except URLGrabError, e: - exception = e - callback = opts.failure_callback -- retrycode = e.errno - except KeyboardInterrupt, e: - exception = e - callback = opts.interrupt_callback -+ if not callback: -+ raise - - if DEBUG: DEBUG.info('exception: %s', exception) - if callback: - if DEBUG: DEBUG.info('calling callback: %s', callback) -- cb_func, cb_args, cb_kwargs = self._make_callback(callback) - obj = CallbackObject(exception=exception, url=args[0], - tries=tries, retry=opts.retry) -- cb_func(obj, *cb_args, **cb_kwargs) -+ _run_callback(callback, obj) - - if (opts.retry is None) or (tries == opts.retry): - if DEBUG: DEBUG.info('retries exceeded, re-raising') - raise - -+ retrycode = getattr(exception, 'errno', None) - if (retrycode is not None) and (retrycode not in opts.retrycodes): - if DEBUG: DEBUG.info('retrycode (%i) not in list %s, re-raising', - retrycode, opts.retrycodes) - raise - -- def urlopen(self, url, **kwargs): -+ def urlopen(self, url, opts=None, **kwargs): - """open the url and return a file object - If a progress object or throttle value specified when this - object was created, then a special file object will be - returned that supports them. The file object can be treated - like any other file object. - """ -- opts = self.opts.derive(**kwargs) -+ url = _to_utf8(url) -+ opts = (opts or self.opts).derive(**kwargs) - if DEBUG: DEBUG.debug('combined options: %s' % repr(opts)) - (url,parts) = opts.urlparser.parse(url, opts) -+ opts.find_proxy(url, parts[0]) - def retryfunc(opts, url): - return PyCurlFileObject(url, filename=None, opts=opts) - return self._retry(opts, retryfunc, url) - -- def urlgrab(self, url, filename=None, **kwargs): -+ def urlgrab(self, url, filename=None, opts=None, **kwargs): - """grab the file at <url> and make a local copy at <filename> - If filename is none, the basename of the url is used. - urlgrab returns the filename of the local file, which may be - different from the passed-in filename if copy_local == 0. - """ -- opts = self.opts.derive(**kwargs) -+ url = _to_utf8(url) -+ opts = (opts or self.opts).derive(**kwargs) - if DEBUG: DEBUG.debug('combined options: %s' % repr(opts)) - (url,parts) = opts.urlparser.parse(url, opts) - (scheme, host, path, parm, query, frag) = parts -+ opts.find_proxy(url, scheme) - if filename is None: - filename = os.path.basename( urllib.unquote(path) ) -+ if not filename: -+ # This is better than nothing. -+ filename = 'index.html' - if scheme == 'file' and not opts.copy_local: - # just return the name of the local file - don't make a - # copy currently -@@ -950,41 +1122,51 @@ class URLGrabber: - - elif not opts.range: - if not opts.checkfunc is None: -- cb_func, cb_args, cb_kwargs = \ -- self._make_callback(opts.checkfunc) -- obj = CallbackObject() -- obj.filename = path -- obj.url = url -- apply(cb_func, (obj, )+cb_args, cb_kwargs) -+ obj = CallbackObject(filename=path, url=url) -+ _run_callback(opts.checkfunc, obj) - return path - -+ if opts.async: -+ opts.url = url -+ opts.filename = filename -+ opts.size = int(opts.size or 0) -+ _async_queue.append(opts) -+ return filename -+ - def retryfunc(opts, url, filename): - fo = PyCurlFileObject(url, filename, opts) - try: - fo._do_grab() -+ if fo._tm_last: -+ dlsz = fo._tm_last[0] - fo._tm_first[0] -+ dltm = fo._tm_last[1] - fo._tm_first[1] -+ _TH.update(url, dlsz, dltm, None) - if not opts.checkfunc is None: -- cb_func, cb_args, cb_kwargs = \ -- self._make_callback(opts.checkfunc) -- obj = CallbackObject() -- obj.filename = filename -- obj.url = url -- apply(cb_func, (obj, )+cb_args, cb_kwargs) -+ obj = CallbackObject(filename=filename, url=url) -+ _run_callback(opts.checkfunc, obj) - finally: - fo.close() - return filename - -- return self._retry(opts, retryfunc, url, filename) -+ try: -+ return self._retry(opts, retryfunc, url, filename) -+ except URLGrabError, e: -+ _TH.update(url, 0, 0, e) -+ opts.exception = e -+ return _run_callback(opts.failfunc, opts) - -- def urlread(self, url, limit=None, **kwargs): -+ def urlread(self, url, limit=None, opts=None, **kwargs): - """read the url into a string, up to 'limit' bytes - If the limit is exceeded, an exception will be thrown. Note - that urlread is NOT intended to be used as a way of saying - "I want the first N bytes" but rather 'read the whole file - into memory, but don't use too much' - """ -- opts = self.opts.derive(**kwargs) -+ url = _to_utf8(url) -+ opts = (opts or self.opts).derive(**kwargs) - if DEBUG: DEBUG.debug('combined options: %s' % repr(opts)) - (url,parts) = opts.urlparser.parse(url, opts) -+ opts.find_proxy(url, parts[0]) - if limit is not None: - limit = limit + 1 - -@@ -1000,12 +1182,8 @@ class URLGrabber: - else: s = fo.read(limit) - - if not opts.checkfunc is None: -- cb_func, cb_args, cb_kwargs = \ -- self._make_callback(opts.checkfunc) -- obj = CallbackObject() -- obj.data = s -- obj.url = url -- apply(cb_func, (obj, )+cb_args, cb_kwargs) -+ obj = CallbackObject(data=s, url=url) -+ _run_callback(opts.checkfunc, obj) - finally: - fo.close() - return s -@@ -1020,6 +1198,7 @@ class URLGrabber: - return s - - def _make_callback(self, callback_obj): -+ # not used, left for compatibility - if callable(callback_obj): - return callback_obj, (), {} - else: -@@ -1030,7 +1209,7 @@ class URLGrabber: - default_grabber = URLGrabber() - - --class PyCurlFileObject(): -+class PyCurlFileObject(object): - def __init__(self, url, filename, opts): - self.fo = None - self._hdr_dump = '' -@@ -1052,10 +1231,13 @@ class PyCurlFileObject(): - self._reget_length = 0 - self._prog_running = False - self._error = (None, None) -- self.size = None -+ self.size = 0 -+ self._hdr_ended = False -+ self._tm_first = None -+ self._tm_last = None - self._do_open() - -- -+ - def __getattr__(self, name): - """This effectively allows us to wrap at the instance level. - Any attribute not found in _this_ object will be searched for -@@ -1067,6 +1249,12 @@ class PyCurlFileObject(): - - def _retrieve(self, buf): - try: -+ tm = self._amount_read + len(buf), time.time() -+ if self._tm_first is None: -+ self._tm_first = tm -+ else: -+ self._tm_last = tm -+ - if not self._prog_running: - if self.opts.progress_obj: - size = self.size + self._reget_length -@@ -1079,15 +1267,24 @@ class PyCurlFileObject(): - self.opts.progress_obj.update(self._amount_read) - - self._amount_read += len(buf) -- self.fo.write(buf) -+ try: -+ self.fo.write(buf) -+ except IOError, e: -+ self._cb_error = URLGrabError(16, exception2msg(e)) -+ return -1 - return len(buf) - except KeyboardInterrupt: - return -1 - - def _hdr_retrieve(self, buf): -+ if self._hdr_ended: -+ self._hdr_dump = '' -+ self.size = 0 -+ self._hdr_ended = False -+ - if self._over_max_size(cur=len(self._hdr_dump), - max_size=self.opts.max_header_size): -- return -1 -+ return -1 - try: - self._hdr_dump += buf - # we have to get the size before we do the progress obj start -@@ -1104,7 +1301,17 @@ class PyCurlFileObject(): - s = parse150(buf) - if s: - self.size = int(s) -- -+ -+ if buf.lower().find('location') != -1: -+ location = ':'.join(buf.split(':')[1:]) -+ location = location.strip() -+ self.scheme = urlparse.urlsplit(location)[0] -+ self.url = location -+ -+ if len(self._hdr_dump) != 0 and buf == '\r\n': -+ self._hdr_ended = True -+ if DEBUG: DEBUG.debug('header ended:') -+ - return len(buf) - except KeyboardInterrupt: - return pycurl.READFUNC_ABORT -@@ -1113,8 +1320,10 @@ class PyCurlFileObject(): - if self._parsed_hdr: - return self._parsed_hdr - statusend = self._hdr_dump.find('\n') -+ statusend += 1 # ridiculous as it may seem. - hdrfp = StringIO() - hdrfp.write(self._hdr_dump[statusend:]) -+ hdrfp.seek(0) - self._parsed_hdr = mimetools.Message(hdrfp) - return self._parsed_hdr - -@@ -1127,6 +1336,9 @@ class PyCurlFileObject(): - if not opts: - opts = self.opts - -+ # keepalives -+ if not opts.keepalive: -+ self.curl_obj.setopt(pycurl.FORBID_REUSE, 1) - - # defaults we're always going to set - self.curl_obj.setopt(pycurl.NOPROGRESS, False) -@@ -1136,11 +1348,21 @@ class PyCurlFileObject(): - self.curl_obj.setopt(pycurl.PROGRESSFUNCTION, self._progress_update) - self.curl_obj.setopt(pycurl.FAILONERROR, True) - self.curl_obj.setopt(pycurl.OPT_FILETIME, True) -+ self.curl_obj.setopt(pycurl.FOLLOWLOCATION, True) - -- if DEBUG: -+ if DEBUG and DEBUG.level <= 10: - self.curl_obj.setopt(pycurl.VERBOSE, True) - if opts.user_agent: - self.curl_obj.setopt(pycurl.USERAGENT, opts.user_agent) -+ if opts.ip_resolve: -+ # Default is: IPRESOLVE_WHATEVER -+ ipr = opts.ip_resolve.lower() -+ if ipr == 'whatever': # Do we need this? -+ self.curl_obj.setopt(pycurl.IPRESOLVE,pycurl.IPRESOLVE_WHATEVER) -+ if ipr == 'ipv4': -+ self.curl_obj.setopt(pycurl.IPRESOLVE, pycurl.IPRESOLVE_V4) -+ if ipr == 'ipv6': -+ self.curl_obj.setopt(pycurl.IPRESOLVE, pycurl.IPRESOLVE_V6) - - # maybe to be options later - self.curl_obj.setopt(pycurl.FOLLOWLOCATION, True) -@@ -1148,9 +1370,11 @@ class PyCurlFileObject(): - - # timeouts - timeout = 300 -- if opts.timeout: -- timeout = int(opts.timeout) -- self.curl_obj.setopt(pycurl.CONNECTTIMEOUT, timeout) -+ if hasattr(opts, 'timeout'): -+ timeout = int(opts.timeout or 0) -+ self.curl_obj.setopt(pycurl.CONNECTTIMEOUT, timeout) -+ self.curl_obj.setopt(pycurl.LOW_SPEED_LIMIT, 1000) -+ self.curl_obj.setopt(pycurl.LOW_SPEED_TIME, timeout) - - # ssl options - if self.scheme == 'https': -@@ -1158,13 +1382,16 @@ class PyCurlFileObject(): - self.curl_obj.setopt(pycurl.CAPATH, opts.ssl_ca_cert) - self.curl_obj.setopt(pycurl.CAINFO, opts.ssl_ca_cert) - self.curl_obj.setopt(pycurl.SSL_VERIFYPEER, opts.ssl_verify_peer) -- self.curl_obj.setopt(pycurl.SSL_VERIFYHOST, opts.ssl_verify_host) -+ if opts.ssl_verify_host: # 1 is meaningless to curl -+ self.curl_obj.setopt(pycurl.SSL_VERIFYHOST, 2) - if opts.ssl_key: - self.curl_obj.setopt(pycurl.SSLKEY, opts.ssl_key) - if opts.ssl_key_type: - self.curl_obj.setopt(pycurl.SSLKEYTYPE, opts.ssl_key_type) - if opts.ssl_cert: - self.curl_obj.setopt(pycurl.SSLCERT, opts.ssl_cert) -+ # if we have a client side cert - turn off reuse b/c nss is odd -+ self.curl_obj.setopt(pycurl.FORBID_REUSE, 1) - if opts.ssl_cert_type: - self.curl_obj.setopt(pycurl.SSLCERTTYPE, opts.ssl_cert_type) - if opts.ssl_key_pass: -@@ -1187,28 +1414,26 @@ class PyCurlFileObject(): - if hasattr(opts, 'raw_throttle') and opts.raw_throttle(): - self.curl_obj.setopt(pycurl.MAX_RECV_SPEED_LARGE, int(opts.raw_throttle())) - -- # proxy settings -- if opts.proxies: -- for (scheme, proxy) in opts.proxies.items(): -- if self.scheme in ('ftp'): # only set the ftp proxy for ftp items -- if scheme not in ('ftp'): -- continue -- else: -- if proxy == '_none_': proxy = "" -- self.curl_obj.setopt(pycurl.PROXY, proxy) -- elif self.scheme in ('http', 'https'): -- if scheme not in ('http', 'https'): -- continue -- else: -- if proxy == '_none_': proxy = "" -- self.curl_obj.setopt(pycurl.PROXY, proxy) -- -- # FIXME username/password/auth settings -+ # proxy -+ if opts.proxy is not None: -+ self.curl_obj.setopt(pycurl.PROXY, opts.proxy) -+ self.curl_obj.setopt(pycurl.PROXYAUTH, -+ # All but Kerberos. BZ 769254 -+ pycurl.HTTPAUTH_ANY - pycurl.HTTPAUTH_GSSNEGOTIATE) -+ -+ if opts.username and opts.password: -+ if self.scheme in ('http', 'https'): -+ self.curl_obj.setopt(pycurl.HTTPAUTH, pycurl.HTTPAUTH_ANY) -+ -+ if opts.username and opts.password: -+ # apparently when applying them as curlopts they do not require quoting of any kind -+ userpwd = '%s:%s' % (opts.username, opts.password) -+ self.curl_obj.setopt(pycurl.USERPWD, userpwd) - - #posts - simple - expects the fields as they are - if opts.data: - self.curl_obj.setopt(pycurl.POST, True) -- self.curl_obj.setopt(pycurl.POSTFIELDS, self._to_utf8(opts.data)) -+ self.curl_obj.setopt(pycurl.POSTFIELDS, _to_utf8(opts.data)) - - # our url - self.curl_obj.setopt(pycurl.URL, self.url) -@@ -1228,39 +1453,36 @@ class PyCurlFileObject(): - - code = self.http_code - errcode = e.args[0] -+ errurl = urllib.unquote(self.url) -+ - if self._error[0]: - errcode = self._error[0] - -- if errcode == 23 and code >= 200 and code < 299: -- err = URLGrabError(15, _('User (or something) called abort %s: %s') % (self.url, e)) -- err.url = self.url -- -+ if errcode == 23 and 200 <= code <= 299: - # this is probably wrong but ultimately this is what happens - # we have a legit http code and a pycurl 'writer failed' code - # which almost always means something aborted it from outside - # since we cannot know what it is -I'm banking on it being - # a ctrl-c. XXXX - if there's a way of going back two raises to - # figure out what aborted the pycurl process FIXME -- raise KeyboardInterrupt -+ raise getattr(self, '_cb_error', KeyboardInterrupt) - - elif errcode == 28: -- err = URLGrabError(12, _('Timeout on %s: %s') % (self.url, e)) -- err.url = self.url -+ err = URLGrabError(12, _('Timeout on %s: %s') % (errurl, e)) -+ err.url = errurl - raise err - elif errcode == 35: - msg = _("problem making ssl connection") - err = URLGrabError(14, msg) -- err.url = self.url -+ err.url = errurl - raise err - elif errcode == 37: -- msg = _("Could not open/read %s") % (self.url) -+ msg = _("Could not open/read %s") % (errurl) - err = URLGrabError(14, msg) -- err.url = self.url -+ err.url = errurl - raise err - - elif errcode == 42: -- err = URLGrabError(15, _('User (or something) called abort %s: %s') % (self.url, e)) -- err.url = self.url - # this is probably wrong but ultimately this is what happens - # we have a legit http code and a pycurl 'writer failed' code - # which almost always means something aborted it from outside -@@ -1272,33 +1494,94 @@ class PyCurlFileObject(): - elif errcode == 58: - msg = _("problem with the local client certificate") - err = URLGrabError(14, msg) -- err.url = self.url -+ err.url = errurl - raise err - - elif errcode == 60: -- msg = _("client cert cannot be verified or client cert incorrect") -+ msg = _("Peer cert cannot be verified or peer cert invalid") - err = URLGrabError(14, msg) -- err.url = self.url -+ err.url = errurl - raise err - - elif errcode == 63: - if self._error[1]: - msg = self._error[1] - else: -- msg = _("Max download size exceeded on %s") % (self.url) -+ msg = _("Max download size exceeded on %s") % () - err = URLGrabError(14, msg) -- err.url = self.url -+ err.url = errurl - raise err - -- elif str(e.args[1]) == '' and self.http_code != 0: # fake it until you make it -- msg = 'HTTP Error %s : %s ' % (self.http_code, self.url) -+ elif str(e.args[1]) == '' and code and not 200 <= code <= 299: -+ if self.scheme in ['http', 'https']: -+ if self.http_code in responses: -+ resp = responses[self.http_code] -+ msg = 'HTTP Error %s - %s : %s' % (self.http_code, resp, errurl) -+ else: -+ msg = 'HTTP Error %s : %s ' % (self.http_code, errurl) -+ elif self.scheme in ['ftp']: -+ msg = 'FTP Error %s : %s ' % (self.http_code, errurl) -+ else: -+ msg = "Unknown Error: URL=%s , scheme=%s" % (errurl, self.scheme) - else: -- msg = 'PYCURL ERROR %s - "%s"' % (errcode, str(e.args[1])) -+ pyerr2str = { 5 : _("Couldn't resolve proxy"), -+ 6 : _("Couldn't resolve host"), -+ 7 : _("Couldn't connect"), -+ 8 : _("Bad reply to FTP server"), -+ 9 : _("Access denied"), -+ 11 : _("Bad reply to FTP pass"), -+ 13 : _("Bad reply to FTP pasv"), -+ 14 : _("Bad reply to FTP 227"), -+ 15 : _("Couldn't get FTP host"), -+ 17 : _("Couldn't set FTP type"), -+ 18 : _("Partial file"), -+ 19 : _("FTP RETR command failed"), -+ 22 : _("HTTP returned error"), -+ 23 : _("Write error"), -+ 25 : _("Upload failed"), -+ 26 : _("Read error"), -+ 27 : _("Out of Memory"), -+ 28 : _("Operation timed out"), -+ 30 : _("FTP PORT command failed"), -+ 31 : _("FTP REST command failed"), -+ 33 : _("Range failed"), -+ 34 : _("HTTP POST failed"), -+ 35 : _("SSL CONNECT failed"), -+ 36 : _("Couldn't resume download"), -+ 37 : _("Couldn't read file"), -+ 42 : _("Aborted by callback"), -+ 47 : _("Too many redirects"), -+ 51 : _("Peer certificate failed verification"), -+ 52 : _("Got nothing: SSL certificate expired?"), -+ 53 : _("SSL engine not found"), -+ 54 : _("SSL engine set failed"), -+ 55 : _("Network error send()"), -+ 56 : _("Network error recv()"), -+ 58 : _("Local certificate failed"), -+ 59 : _("SSL set cipher failed"), -+ 60 : _("Local CA certificate failed"), -+ 61 : _("HTTP bad transfer encoding"), -+ 63 : _("Maximum file size exceeded"), -+ 64 : _("FTP SSL failed"), -+ 67 : _("Authentication failure"), -+ 70 : _("Out of disk space on server"), -+ 73 : _("Remove file exists"), -+ } -+ errstr = str(e.args[1]) -+ if not errstr: -+ errstr = pyerr2str.get(errcode, '<Unknown>') -+ msg = 'curl#%s - "%s"' % (errcode, errstr) - code = errcode - err = URLGrabError(14, msg) - err.code = code - err.exception = e - raise err -+ else: -+ if self._error[1]: -+ msg = self._error[1] -+ err = URLGrabError(14, msg) -+ err.url = urllib.unquote(self.url) -+ raise err - - def _do_open(self): - self.curl_obj = _curl_cache -@@ -1333,7 +1616,11 @@ class PyCurlFileObject(): - - if self.opts.range: - rt = self.opts.range -- if rt[0]: rt = (rt[0] + reget_length, rt[1]) -+ -+ if rt[0] is None: -+ rt = (0, rt[1]) -+ rt = (rt[0] + reget_length, rt[1]) -+ - - if rt: - header = range_tuple_to_header(rt) -@@ -1434,21 +1721,46 @@ class PyCurlFileObject(): - #fh, self._temp_name = mkstemp() - #self.fo = open(self._temp_name, 'wb') - -- -- self._do_perform() -- -- -- -+ try: -+ self._do_perform() -+ except URLGrabError, e: -+ self.fo.flush() -+ self.fo.close() -+ raise e -+ - if _was_filename: - # close it up - self.fo.flush() - self.fo.close() -+ -+ # Set the URL where we got it from: -+ if xattr is not None: -+ # See: http://www.freedesktop.org/wiki/CommonExtendedAttributes -+ try: -+ xattr.set(self.filename, 'user.xdg.origin.url', self.url) -+ except: -+ pass # URL too long. = IOError ... ignore everything. -+ - # set the time - mod_time = self.curl_obj.getinfo(pycurl.INFO_FILETIME) - if mod_time != -1: -- os.utime(self.filename, (mod_time, mod_time)) -+ try: -+ os.utime(self.filename, (mod_time, mod_time)) -+ except OSError, e: -+ err = URLGrabError(16, _(\ -+ 'error setting timestamp on file %s from %s, OSError: %s') -+ % (self.filename, self.url, e)) -+ err.url = self.url -+ raise err - # re open it -- self.fo = open(self.filename, 'r') -+ try: -+ self.fo = open(self.filename, 'r') -+ except IOError, e: -+ err = URLGrabError(16, _(\ -+ 'error opening file from %s, IOError: %s') % (self.url, e)) -+ err.url = self.url -+ raise err -+ - else: - #self.fo = open(self._temp_name, 'r') - self.fo.seek(0) -@@ -1526,17 +1838,20 @@ class PyCurlFileObject(): - if self._prog_running: - downloaded += self._reget_length - self.opts.progress_obj.update(downloaded) -- except KeyboardInterrupt: -+ except (KeyboardInterrupt, IOError): - return -1 - - def _over_max_size(self, cur, max_size=None): - - if not max_size: -- max_size = self.size -- if self.opts.size: # if we set an opts size use that, no matter what -- max_size = self.opts.size -+ if not self.opts.size: -+ max_size = self.size -+ else: -+ max_size = self.opts.size -+ - if not max_size: return False # if we have None for all of the Max then this is dumb -- if cur > max_size + max_size*.10: -+ -+ if cur > int(float(max_size) * 1.10): - - msg = _("Downloaded more than max size for %s: %s > %s") \ - % (self.url, cur, max_size) -@@ -1544,13 +1859,6 @@ class PyCurlFileObject(): - return True - return False - -- def _to_utf8(self, obj, errors='replace'): -- '''convert 'unicode' to an encoded utf-8 byte string ''' -- # stolen from yum.i18n -- if isinstance(obj, unicode): -- obj = obj.encode('utf-8', errors) -- return obj -- - def read(self, amt=None): - self._fill_buffer(amt) - if amt is None: -@@ -1582,9 +1890,21 @@ class PyCurlFileObject(): - self.opts.progress_obj.end(self._amount_read) - self.fo.close() - -- -+ def geturl(self): -+ """ Provide the geturl() method, used to be got from -+ urllib.addinfourl, via. urllib.URLopener.* """ -+ return self.url -+ - _curl_cache = pycurl.Curl() # make one and reuse it over and over and over - -+def reset_curl_obj(): -+ """To make sure curl has reread the network/dns info we force a reload""" -+ global _curl_cache -+ _curl_cache.close() -+ _curl_cache = pycurl.Curl() -+ -+_libproxy_cache = None -+ - - ##################################################################### - # DEPRECATED FUNCTIONS -@@ -1621,6 +1941,478 @@ def retrygrab(url, filename=None, copy_local=0, close_connection=0, - - - ##################################################################### -+# Serializer + parser: A replacement of the rather bulky Json code. -+# -+# - handles basic python literals, lists and tuples. -+# - serialized strings never contain ' ' or '\n' -+# -+##################################################################### -+ -+_quoter_map = {} -+for c in '%[(,)] \n': -+ _quoter_map[c] = '%%%02x' % ord(c) -+del c -+ -+def _dumps(v): -+ if v is None: return 'None' -+ if v is True: return 'True' -+ if v is False: return 'False' -+ if type(v) in (int, long, float): -+ return str(v) -+ if type(v) == unicode: -+ v = v.encode('UTF8') -+ if type(v) == str: -+ def quoter(c): return _quoter_map.get(c, c) -+ return "'%s'" % ''.join(map(quoter, v)) -+ if type(v) == tuple: -+ return "(%s)" % ','.join(map(_dumps, v)) -+ if type(v) == list: -+ return "[%s]" % ','.join(map(_dumps, v)) -+ raise TypeError, 'Can\'t serialize %s' % v -+ -+def _loads(s): -+ def decode(v): -+ if v == 'None': return None -+ if v == 'True': return True -+ if v == 'False': return False -+ try: return int(v) -+ except ValueError: pass -+ try: return float(v) -+ except ValueError: pass -+ if len(v) >= 2 and v[0] == v[-1] == "'": -+ ret = []; i = 1 -+ while True: -+ j = v.find('%', i) -+ ret.append(v[i:j]) # skips the final "'" -+ if j == -1: break -+ ret.append(chr(int(v[j + 1:j + 3], 16))) -+ i = j + 3 -+ v = ''.join(ret) -+ return v -+ stk = None -+ l = [] -+ i = j = 0 -+ while True: -+ if j == len(s) or s[j] in ',)]': -+ if j > i: -+ l.append(decode(s[i:j])) -+ if j == len(s): break -+ if s[j] in ')]': -+ if s[j] == ')': -+ l = tuple(l) -+ stk[0].append(l) -+ l, stk = stk -+ i = j = j + 1 -+ elif s[j] in '[(': -+ stk = l, stk -+ l = [] -+ i = j = j + 1 -+ else: -+ j += 1 # safe because '[(,)]' are quoted -+ if stk: raise ValueError -+ if len(l) == 1: l = l[0] -+ return l -+ -+ -+##################################################################### -+# External downloader process -+##################################################################### -+ -+def _readlines(fd): -+ buf = os.read(fd, 4096) -+ if not buf: return None -+ # whole lines only, no buffering -+ while buf[-1] != '\n': -+ buf += os.read(fd, 4096) -+ return buf[:-1].split('\n') -+ -+import subprocess -+ -+class _ExternalDownloader: -+ def __init__(self): -+ self.popen = subprocess.Popen( -+ '/usr/libexec/urlgrabber-ext-down', -+ stdin = subprocess.PIPE, -+ stdout = subprocess.PIPE, -+ ) -+ self.stdin = self.popen.stdin.fileno() -+ self.stdout = self.popen.stdout.fileno() -+ self.running = {} -+ self.cnt = 0 -+ -+ # list of options we pass to downloader -+ _options = ( -+ 'url', 'filename', -+ 'timeout', 'close_connection', 'keepalive', -+ 'throttle', 'bandwidth', 'range', 'reget', -+ 'user_agent', 'http_headers', 'ftp_headers', -+ 'proxy', 'prefix', 'username', 'password', -+ 'ssl_ca_cert', -+ 'ssl_cert', 'ssl_cert_type', -+ 'ssl_key', 'ssl_key_type', -+ 'ssl_key_pass', -+ 'ssl_verify_peer', 'ssl_verify_host', -+ 'size', 'max_header_size', 'ip_resolve', -+ ) -+ -+ def start(self, opts): -+ arg = [] -+ for k in self._options: -+ v = getattr(opts, k) -+ if v is None: continue -+ arg.append('%s=%s' % (k, _dumps(v))) -+ if opts.progress_obj and opts.multi_progress_obj: -+ arg.append('progress_obj=True') -+ arg = ' '.join(arg) -+ if DEBUG: DEBUG.info('attempt %i/%s: %s', opts.tries, opts.retry, opts.url) -+ -+ self.cnt += 1 -+ self.running[self.cnt] = opts -+ os.write(self.stdin, arg +'\n') -+ -+ def perform(self): -+ ret = [] -+ lines = _readlines(self.stdout) -+ if not lines: -+ if DEBUG: DEBUG.info('downloader died') -+ raise KeyboardInterrupt -+ for line in lines: -+ # parse downloader output -+ line = line.split(' ', 5) -+ _id, size = map(int, line[:2]) -+ if len(line) == 2: -+ self.running[_id]._progress.update(size) -+ continue -+ # job done -+ opts = self.running.pop(_id) -+ if line[4] == 'OK': -+ ug_err = None -+ if DEBUG: DEBUG.info('success') -+ else: -+ ug_err = URLGrabError(int(line[4]), line[5]) -+ if DEBUG: DEBUG.info('failure: %s', ug_err) -+ _TH.update(opts.url, int(line[2]), float(line[3]), ug_err, opts.async[0]) -+ ret.append((opts, size, ug_err)) -+ return ret -+ -+ def abort(self): -+ self.popen.stdin.close() -+ self.popen.stdout.close() -+ self.popen.wait() -+ -+class _ExternalDownloaderPool: -+ def __init__(self): -+ self.epoll = select.epoll() -+ self.running = {} -+ self.cache = {} -+ -+ def start(self, opts): -+ host = urlparse.urlsplit(opts.url).netloc -+ dl = self.cache.pop(host, None) -+ if not dl: -+ dl = _ExternalDownloader() -+ fl = fcntl.fcntl(dl.stdin, fcntl.F_GETFD) -+ fcntl.fcntl(dl.stdin, fcntl.F_SETFD, fl | fcntl.FD_CLOEXEC) -+ self.epoll.register(dl.stdout, select.EPOLLIN) -+ self.running[dl.stdout] = dl -+ dl.start(opts) -+ -+ def perform(self): -+ ret = [] -+ for fd, event in self.epoll.poll(): -+ if event & select.EPOLLHUP: -+ if DEBUG: DEBUG.info('downloader died') -+ raise KeyboardInterrupt -+ assert event & select.EPOLLIN -+ done = self.running[fd].perform() -+ if not done: continue -+ assert len(done) == 1 -+ ret.extend(done) -+ -+ # dl finished, move it to the cache -+ host = urlparse.urlsplit(done[0][0].url).netloc -+ if host in self.cache: self.cache[host].abort() -+ self.epoll.unregister(fd) -+ self.cache[host] = self.running.pop(fd) -+ return ret -+ -+ def abort(self): -+ for dl in self.running.values(): -+ self.epoll.unregister(dl.stdout) -+ dl.abort() -+ for dl in self.cache.values(): -+ dl.abort() -+ -+ -+##################################################################### -+# High level async API -+##################################################################### -+ -+_async_queue = [] -+ -+def parallel_wait(meter=None): -+ '''Process queued requests in parallel. -+ ''' -+ -+ # calculate total sizes -+ meters = {} -+ for opts in _async_queue: -+ if opts.progress_obj and opts.multi_progress_obj: -+ count, total = meters.get(opts.multi_progress_obj) or (0, 0) -+ meters[opts.multi_progress_obj] = count + 1, total + opts.size -+ -+ # start multi-file meters -+ for meter in meters: -+ count, total = meters[meter] -+ meter.start(count, total) -+ -+ dl = _ExternalDownloaderPool() -+ host_con = {} # current host connection counts -+ -+ def start(opts, tries): -+ opts.tries = tries -+ try: -+ dl.start(opts) -+ except OSError, e: -+ # can't spawn downloader, give up immediately -+ opts.exception = URLGrabError(5, exception2msg(e)) -+ _run_callback(opts.failfunc, opts) -+ return -+ -+ key, limit = opts.async -+ host_con[key] = host_con.get(key, 0) + 1 -+ if opts.progress_obj: -+ if opts.multi_progress_obj: -+ opts._progress = opts.multi_progress_obj.newMeter() -+ opts._progress.start(text=opts.text) -+ else: -+ opts._progress = time.time() # no updates -+ -+ def perform(): -+ for opts, size, ug_err in dl.perform(): -+ key, limit = opts.async -+ host_con[key] -= 1 -+ -+ if ug_err is None: -+ if opts.checkfunc: -+ try: _run_callback(opts.checkfunc, opts) -+ except URLGrabError, ug_err: pass -+ -+ if opts.progress_obj: -+ if opts.multi_progress_obj: -+ if ug_err: -+ opts._progress.failure(None) -+ else: -+ opts.multi_progress_obj.re.total += size - opts.size # correct totals -+ opts._progress.end(size) -+ opts.multi_progress_obj.removeMeter(opts._progress) -+ else: -+ opts.progress_obj.start(text=opts.text, now=opts._progress) -+ opts.progress_obj.update(size) -+ opts.progress_obj.end(size) -+ del opts._progress -+ -+ if ug_err is None: -+ continue -+ -+ retry = opts.retry or 0 -+ if opts.failure_callback: -+ opts.exception = ug_err -+ try: _run_callback(opts.failure_callback, opts) -+ except URLGrabError, ug_err: -+ retry = 0 # no retries -+ if opts.tries < retry and ug_err.errno in opts.retrycodes: -+ start(opts, opts.tries + 1) # simple retry -+ continue -+ -+ if opts.mirror_group: -+ mg, errors, failed, removed = opts.mirror_group -+ errors.append((opts.url, exception2msg(ug_err))) -+ failed[key] = failed.get(key, 0) + 1 -+ opts.mirror = key -+ opts.exception = ug_err -+ action = mg.default_action or {} -+ if mg.failure_callback: -+ opts.tries = len(errors) -+ action = dict(action) # update only the copy -+ action.update(_run_callback(mg.failure_callback, opts)) -+ if not action.get('fail', 0): -+ # mask this mirror and retry -+ if action.get('remove', 1): -+ removed.add(key) -+ _async_queue.append(opts) -+ continue -+ # fail=1 from callback -+ ug_err.errors = errors -+ -+ # urlgrab failed -+ opts.exception = ug_err -+ _run_callback(opts.failfunc, opts) -+ -+ try: -+ idx = 0 -+ while True: -+ if idx >= len(_async_queue): -+ # the queue is empty -+ if not dl.running: break -+ # pending dl may extend it -+ perform() -+ continue -+ -+ # handle next request -+ opts = _async_queue[idx] -+ idx += 1 -+ -+ # check global limit -+ while len(dl.running) >= default_grabber.opts.max_connections: -+ perform() -+ if DEBUG: -+ DEBUG.info('max_connections: %d/%d', len(dl.running), default_grabber.opts.max_connections) -+ -+ if opts.mirror_group: -+ mg, errors, failed, removed = opts.mirror_group -+ -+ # find the best mirror -+ best = None -+ best_speed = None -+ for mirror in mg.mirrors: -+ key = mirror['mirror'] -+ if key in removed: continue -+ -+ # estimate mirror speed -+ speed, fail = _TH.estimate(key) -+ speed /= 1 + host_con.get(key, 0) -+ -+ # order by: least failures, private flag, best speed -+ # ignore 'private' flag if there were failures -+ private = not fail and mirror.get('kwargs', {}).get('private', False) -+ speed = -failed.get(key, 0), private, speed -+ if best is None or speed > best_speed: -+ best = mirror -+ best_speed = speed -+ -+ if best is None: -+ opts.exception = URLGrabError(256, _('No more mirrors to try.')) -+ opts.exception.errors = errors -+ _run_callback(opts.failfunc, opts) -+ continue -+ -+ # update the grabber object, apply mirror kwargs -+ grabber = best.get('grabber') or mg.grabber -+ opts.delegate = grabber.opts.derive(**best.get('kwargs', {})) -+ -+ # update the current mirror and limit -+ key = best['mirror'] -+ limit = best.get('kwargs', {}).get('max_connections', 2) -+ opts.async = key, limit -+ -+ # update URL and proxy -+ url = mg._join_url(key, opts.relative_url) -+ url, parts = opts.urlparser.parse(url, opts) -+ opts.find_proxy(url, parts[0]) -+ opts.url = url -+ -+ # check host limit, then start -+ key, limit = opts.async -+ while host_con.get(key, 0) >= limit: -+ perform() -+ if DEBUG: -+ DEBUG.info('max_connections(%s): %d/%d', key, host_con.get(key, 0), limit) -+ -+ start(opts, 1) -+ except IOError, e: -+ if e.errno != 4: raise -+ raise KeyboardInterrupt -+ -+ finally: -+ dl.abort() -+ for meter in meters: -+ meter.end() -+ del _async_queue[:] -+ _TH.save() -+ -+ -+##################################################################### -+# Host bandwidth estimation -+##################################################################### -+ -+class _TH: -+ hosts = {} -+ dirty = None -+ -+ @staticmethod -+ def load(): -+ filename = default_grabber.opts.timedhosts -+ if filename and _TH.dirty is None: -+ try: -+ for line in open(filename): -+ host, speed, fail, ts = line.rsplit(' ', 3) -+ _TH.hosts[host] = int(speed), int(fail), int(ts) -+ except IOError: pass -+ _TH.dirty = False -+ -+ @staticmethod -+ def save(): -+ filename = default_grabber.opts.timedhosts -+ if filename and _TH.dirty is True: -+ tmp = '%s.%d' % (filename, os.getpid()) -+ try: -+ f = open(tmp, 'w') -+ for host in _TH.hosts: -+ f.write(host + ' %d %d %d\n' % _TH.hosts[host]) -+ f.close() -+ os.rename(tmp, filename) -+ except IOError: pass -+ _TH.dirty = False -+ -+ @staticmethod -+ def update(url, dl_size, dl_time, ug_err, baseurl=None): -+ # Use hostname from URL. If it's a file:// URL, use baseurl. -+ # If no baseurl, do not update timedhosts. -+ host = urlparse.urlsplit(url).netloc.split('@')[-1] or baseurl -+ if not host: return -+ -+ _TH.load() -+ speed, fail, ts = _TH.hosts.get(host) or (0, 0, 0) -+ now = time.time() -+ -+ if ug_err is None: -+ # defer first update if the file was small. BZ 851178. -+ if not ts and dl_size < 1e6: return -+ -+ # k1: the older, the less useful -+ # k2: <500ms readings are less reliable -+ # speeds vary, use 10:1 smoothing -+ k1 = 2**((ts - now) / default_grabber.opts.half_life) -+ k2 = min(dl_time / .500, 1.0) / 10 -+ if k2 > 0: -+ speed = (k1 * speed + k2 * dl_size / dl_time) / (k1 + k2) -+ fail = 0 -+ elif getattr(ug_err, 'code', None) == 404: -+ fail = 0 # alive, at least -+ else: -+ fail += 1 # seems dead -+ -+ _TH.hosts[host] = speed, fail, now -+ _TH.dirty = True -+ -+ @staticmethod -+ def estimate(baseurl): -+ _TH.load() -+ -+ # Use just the hostname, unless it's a file:// baseurl. -+ host = urlparse.urlsplit(baseurl).netloc.split('@')[-1] or baseurl -+ -+ default_speed = default_grabber.opts.default_speed -+ try: speed, fail, ts = _TH.hosts[host] -+ except KeyError: return default_speed, 0 -+ -+ speed *= 2**-fail -+ k = 2**((ts - time.time()) / default_grabber.opts.half_life) -+ speed = k * speed + (1 - k) * default_speed -+ return speed, fail -+ -+##################################################################### - # TESTING - def _main_test(): - try: url, filename = sys.argv[1:3] -diff --git a/urlgrabber/mirror.py b/urlgrabber/mirror.py -index dad410b..7975f1b 100644 ---- a/urlgrabber/mirror.py -+++ b/urlgrabber/mirror.py -@@ -76,6 +76,9 @@ CUSTOMIZATION - 'grabber' is omitted, the default grabber will be used. If - kwargs are omitted, then (duh) they will not be used. - -+ kwarg 'max_connections' limits the number of concurrent -+ connections to this mirror. -+ - 3) Pass keyword arguments when instantiating the mirror group. - See, for example, the failure_callback argument. - -@@ -87,10 +90,14 @@ CUSTOMIZATION - """ - - -+import sys - import random - import thread # needed for locking to make this threadsafe - --from grabber import URLGrabError, CallbackObject, DEBUG -+from grabber import URLGrabError, CallbackObject, DEBUG, _to_utf8 -+from grabber import _run_callback, _do_raise -+from grabber import exception2msg -+from grabber import _TH - - def _(st): - return st -@@ -126,7 +133,9 @@ class MirrorGroup: - files) - - * if the local list is ever exhausted, a URLGrabError will be -- raised (errno=256, no more mirrors) -+ raised (errno=256, No more mirrors). The 'errors' attribute -+ holds a list of (full_url, errmsg) tuples. This contains -+ all URLs tried and the corresponding error messages. - - OPTIONS - -@@ -153,7 +162,8 @@ class MirrorGroup: - - The 'fail' option will cause immediate failure by re-raising - the exception and no further attempts to get the current -- download. -+ download. As in the "No more mirrors" case, the 'errors' -+ attribute is set in the exception object. - - This dict can be set at instantiation time, - mg = MirrorGroup(grabber, mirrors, default_action={'fail':1}) -@@ -184,6 +194,7 @@ class MirrorGroup: - - obj.exception = < exception that was raised > - obj.mirror = < the mirror that was tried > -+ obj.tries = < the number of mirror tries so far > - obj.relative_url = < url relative to the mirror > - obj.url = < full url that failed > - # .url is just the combination of .mirror -@@ -251,6 +262,17 @@ class MirrorGroup: - self.default_action = None - self._process_kwargs(kwargs) - -+ # use the same algorithm as parallel downloader to initially sort -+ # the mirror list (sort by speed, but prefer live private mirrors) -+ def estimate(m): -+ speed, fail = _TH.estimate(m['mirror']) -+ private = not fail and m.get('kwargs', {}).get('private', False) -+ return private, speed -+ -+ # update the initial order. since sorting is stable, the relative -+ # order of unknown (not used yet) hosts is retained. -+ self.mirrors.sort(key=estimate, reverse=True) -+ - # if these values are found in **kwargs passed to one of the urlXXX - # methods, they will be stripped before getting passed on to the - # grabber -@@ -263,7 +285,8 @@ class MirrorGroup: - def _parse_mirrors(self, mirrors): - parsed_mirrors = [] - for m in mirrors: -- if type(m) == type(''): m = {'mirror': m} -+ if isinstance(m, basestring): -+ m = {'mirror': _to_utf8(m)} - parsed_mirrors.append(m) - return parsed_mirrors - -@@ -280,7 +303,9 @@ class MirrorGroup: - # return a random mirror so that multiple mirrors get used - # even without failures. - if not gr.mirrors: -- raise URLGrabError(256, _('No more mirrors to try.')) -+ e = URLGrabError(256, _('No more mirrors to try.')) -+ e.errors = gr.errors -+ raise e - return gr.mirrors[gr._next] - - def _failure(self, gr, cb_obj): -@@ -307,7 +332,9 @@ class MirrorGroup: - a.update(action) - action = a - self.increment_mirror(gr, action) -- if action and action.get('fail', 0): raise -+ if action and action.get('fail', 0): -+ sys.exc_info()[1].errors = gr.errors -+ raise - - def increment_mirror(self, gr, action={}): - """Tell the mirror object increment the mirror index -@@ -377,35 +404,50 @@ class MirrorGroup: - gr.url = url - gr.kw = dict(kw) - self._load_gr(gr) -+ gr.errors = [] - - for k in self.options: - try: del kw[k] - except KeyError: pass - -+ tries = 0 - while 1: -+ tries += 1 - mirrorchoice = self._get_mirror(gr) - fullurl = self._join_url(mirrorchoice['mirror'], gr.url) -- kwargs = dict(mirrorchoice.get('kwargs', {})) -- kwargs.update(kw) - grabber = mirrorchoice.get('grabber') or self.grabber -+ # apply mirrorchoice kwargs on top of grabber.opts -+ opts = grabber.opts.derive(**mirrorchoice.get('kwargs', {})) - func_ref = getattr(grabber, func) - if DEBUG: DEBUG.info('MIRROR: trying %s -> %s', url, fullurl) - try: -- return func_ref( *(fullurl,), **kwargs ) -+ return func_ref( *(fullurl,), opts=opts, **kw ) - except URLGrabError, e: - if DEBUG: DEBUG.info('MIRROR: failed') -+ gr.errors.append((fullurl, exception2msg(e))) - obj = CallbackObject() - obj.exception = e - obj.mirror = mirrorchoice['mirror'] - obj.relative_url = gr.url - obj.url = fullurl -+ obj.tries = tries - self._failure(gr, obj) - - def urlgrab(self, url, filename=None, **kwargs): - kw = dict(kwargs) - kw['filename'] = filename -+ if kw.get('async'): -+ # enable mirror failovers in async path -+ kw['mirror_group'] = self, [], {}, set() -+ kw['relative_url'] = url -+ else: -+ kw.pop('failfunc', None) - func = 'urlgrab' -- return self._mirror_try(func, url, kw) -+ try: -+ return self._mirror_try(func, url, kw) -+ except URLGrabError, e: -+ obj = CallbackObject(url=url, filename=filename, exception=e, **kwargs) -+ return _run_callback(kwargs.get('failfunc', _do_raise), obj) - - def urlopen(self, url, **kwargs): - kw = dict(kwargs) -diff --git a/urlgrabber/progress.py b/urlgrabber/progress.py -index dd07c6a..077fd99 100644 ---- a/urlgrabber/progress.py -+++ b/urlgrabber/progress.py -@@ -133,8 +133,8 @@ class BaseMeter: - # for a real gui, you probably want to override and put a call - # to your mainloop iteration function here - if now is None: now = time.time() -- if (now >= self.last_update_time + self.update_period) or \ -- not self.last_update_time: -+ if (not self.last_update_time or -+ (now >= self.last_update_time + self.update_period)): - self.re.update(amount_read, now) - self.last_amount_read = amount_read - self.last_update_time = now -@@ -211,6 +211,21 @@ def text_meter_total_size(size, downloaded=0): - # 4. + ( 5, total: 32) - # - -+def _term_add_bar(tl, bar_max_length, pc): -+ blen = bar_max_length -+ bar = '='*int(blen * pc) -+ if (blen * pc) - int(blen * pc) >= 0.5: -+ bar += '-' -+ return tl.add(' [%-*.*s]' % (blen, blen, bar)) -+ -+def _term_add_end(tl, osize, size): -+ if osize is not None: -+ if size > osize: # Is ??? better? Really need something to say < vs >. -+ return tl.add(' !!! '), True -+ elif size != osize: -+ return tl.add(' ... '), True -+ return tl.add(' ' * 5), False -+ - class TextMeter(BaseMeter): - def __init__(self, fo=sys.stderr): - BaseMeter.__init__(self) -@@ -218,7 +233,6 @@ class TextMeter(BaseMeter): - - def _do_update(self, amount_read, now=None): - etime = self.re.elapsed_time() -- fetime = format_time(etime) - fread = format_number(amount_read) - #self.size = None - if self.text is not None: -@@ -234,16 +248,20 @@ class TextMeter(BaseMeter): - - # Include text + ui_rate in minimal - tl = TerminalLine(8, 8+1+8) -+ if tl._llen > 80: -+ use_hours = True # For big screens, make it more readable. -+ else: -+ use_hours = False - ui_size = tl.add(' | %5sB' % fread) - if self.size is None: -- ui_time = tl.add(' %9s' % fetime) -+ ui_time = tl.add(' %9s' % format_time(etime, use_hours)) - ui_end = tl.add(' ' * 5) - ui_rate = tl.add(' %5sB/s' % ave_dl) - out = '%-*.*s%s%s%s%s\r' % (tl.rest(), tl.rest(), text, - ui_rate, ui_size, ui_time, ui_end) - else: - rtime = self.re.remaining_time() -- frtime = format_time(rtime) -+ frtime = format_time(rtime, use_hours) - frac = self.re.fraction_read() - - ui_time = tl.add(' %9s' % frtime) -@@ -259,13 +277,10 @@ class TextMeter(BaseMeter): - ui_rate = tl.add(' %5sB/s' % ave_dl) - # Make text grow a bit before we start growing the bar too - blen = 4 + tl.rest_split(8 + 8 + 4) -- bar = '='*int(blen * frac) -- if (blen * frac) - int(blen * frac) >= 0.5: -- bar += '-' -- ui_bar = tl.add(' [%-*.*s]' % (blen, blen, bar)) -- out = '%-*.*s%s%s%s%s%s%s%s\r' % (tl.rest(), tl.rest(), text, -- ui_sofar_pc, ui_pc, ui_bar, -- ui_rate, ui_size, ui_time, ui_end) -+ ui_bar = _term_add_bar(tl, blen, frac) -+ out = '\r%-*.*s%s%s%s%s%s%s%s\r' % (tl.rest(), tl.rest(), text, -+ ui_sofar_pc, ui_pc, ui_bar, -+ ui_rate,ui_size,ui_time, ui_end) - - self.fo.write(out) - self.fo.flush() -@@ -274,7 +289,6 @@ class TextMeter(BaseMeter): - global _text_meter_total_size - global _text_meter_sofar_size - -- total_time = format_time(self.re.elapsed_time()) - total_size = format_number(amount_read) - if self.text is not None: - text = self.text -@@ -282,14 +296,13 @@ class TextMeter(BaseMeter): - text = self.basename - - tl = TerminalLine(8) -- ui_size = tl.add(' | %5sB' % total_size) -- ui_time = tl.add(' %9s' % total_time) -- not_done = self.size is not None and amount_read != self.size -- if not_done: -- ui_end = tl.add(' ... ') -+ if tl._llen > 80: -+ use_hours = True # For big screens, make it more readable. - else: -- ui_end = tl.add(' ' * 5) -- -+ use_hours = False -+ ui_size = tl.add(' | %5sB' % total_size) -+ ui_time = tl.add(' %9s' % format_time(self.re.elapsed_time(),use_hours)) -+ ui_end, not_done = _term_add_end(tl, self.size, amount_read) - out = '\r%-*.*s%s%s%s\n' % (tl.rest(), tl.rest(), text, - ui_size, ui_time, ui_end) - self.fo.write(out) -@@ -331,12 +344,21 @@ class MultiFileHelper(BaseMeter): - def message(self, message): - self.master.message_meter(self, message) - -+class _FakeLock: -+ def acquire(self): -+ pass -+ def release(self): -+ pass -+ - class MultiFileMeter: - helperclass = MultiFileHelper -- def __init__(self): -+ def __init__(self, threaded=True): - self.meters = [] - self.in_progress_meters = [] -- self._lock = thread.allocate_lock() -+ if threaded: -+ self._lock = thread.allocate_lock() -+ else: -+ self._lock = _FakeLock() - self.update_period = 0.3 # seconds - - self.numfiles = None -@@ -369,6 +391,7 @@ class MultiFileMeter: - - def end(self, now=None): - if now is None: now = time.time() -+ self.re.update(self._amount_read(), now) - self._do_end(now) - - def _do_end(self, now): -@@ -407,8 +430,8 @@ class MultiFileMeter: - def update_meter(self, meter, now): - if not meter in self.meters: - raise ValueError('attempt to use orphaned meter') -- if (now >= self.last_update_time + self.update_period) or \ -- not self.last_update_time: -+ if (not self.last_update_time or -+ (now >= self.last_update_time + self.update_period)): - self.re.update(self._amount_read(), now) - self.last_update_time = now - self._do_update_meter(meter, now) -@@ -466,34 +489,87 @@ class MultiFileMeter: - - - class TextMultiFileMeter(MultiFileMeter): -- def __init__(self, fo=sys.stderr): -+ def __init__(self, fo=sys.stderr, threaded=True): - self.fo = fo -- MultiFileMeter.__init__(self) -+ MultiFileMeter.__init__(self, threaded) -+ self.index_time = self.index = 0 - - # files: ###/### ###% data: ######/###### ###% time: ##:##:##/##:##:## -+# New output, like TextMeter output... -+# update: No size (minimal: 17 chars) -+# ----------------------------------- -+# (<#file>/<#tot files>): <text> <rate> | <current size> <elapsed> -+# 8-48 1 8 3 6 1 7-9 5 -+# -+# update: Size, All files -+# ----------------------- -+# (<#file>/<#tot files>): <text> <pc> <bar> <rate> | <size> <eta time> ETA -+# 8-22 1 3-4 1 6-12 1 8 3 6 1 7-9 1 3 1 -+# end -+# --- -+# <text> | <file size> <file elapsed time> -+# 8-56 3 6 1 9 5 - def _do_update_meter(self, meter, now): - self._lock.acquire() - try: -- format = "files: %3i/%-3i %3i%% data: %6.6s/%-6.6s %3i%% " \ -- "time: %8.8s/%8.8s" - df = self.finished_files - tf = self.numfiles or 1 -- pf = 100 * float(df)/tf + 0.49 -+ # Don't use "percent of files complete" ... -+ # pf = 100 * float(df)/tf + 0.49 - dd = self.re.last_amount_read -- td = self.total_size -+ td = self.re.total - pd = 100 * (self.re.fraction_read() or 0) + 0.49 - dt = self.re.elapsed_time() - rt = self.re.remaining_time() -- if rt is None: tt = None -- else: tt = dt + rt - -- fdd = format_number(dd) + 'B' -- ftd = format_number(td) + 'B' -- fdt = format_time(dt, 1) -- ftt = format_time(tt, 1) -- -- out = '%-79.79s' % (format % (df, tf, pf, fdd, ftd, pd, fdt, ftt)) -- self.fo.write('\r' + out) -+ frac = self.re.fraction_read() or 0 -+ pf = 100 * frac -+ ave_dl = format_number(self.re.average_rate()) -+ -+ # cycle through active meters -+ if now > self.index_time: -+ self.index_time = now + 1.0 -+ self.index += 1 -+ if self.index >= len(self.meters): -+ self.index = 0 -+ meter = self.meters[self.index] -+ text = meter.text or meter.basename -+ if tf > 1: -+ text = '(%u/%u): %s' % (df+1+self.index, tf, text) -+ -+ # Include text + ui_rate in minimal -+ tl = TerminalLine(8, 8+1+8) -+ if tl._llen > 80: -+ use_hours = True # For big screens, make it more readable. -+ time_len = 9 -+ else: -+ use_hours = False -+ time_len = 7 -+ -+ ui_size = tl.add(' | %5sB' % format_number(dd)) -+ -+ if not self.re.total: -+ ui_time = tl.add(' %*s' % (time_len,format_time(dt, use_hours))) -+ ui_end = tl.add(' ' * 5) -+ ui_rate = tl.add(' %5sB/s' % ave_dl) -+ out = '\r%-*.*s%s%s%s%s\r' % (tl.rest(), tl.rest(), text, -+ ui_rate, ui_size, ui_time, ui_end) -+ else: -+ ui_time = tl.add(' %*s' % (time_len,format_time(rt, use_hours))) -+ ui_end = tl.add(' ETA ') -+ -+ ui_sofar_pc = tl.add(' %i%%' % pf, -+ full_len=len(" (100%)")) -+ ui_rate = tl.add(' %5sB/s' % ave_dl) -+ -+ # Make text grow a bit before we start growing the bar too -+ blen = 4 + tl.rest_split(8 + 8 + 4) -+ ui_bar = _term_add_bar(tl, blen, frac) -+ out = '\r%-*.*s%s%s%s%s%s%s\r' % (tl.rest(), tl.rest(), text, -+ ui_sofar_pc, ui_bar, -+ ui_rate, ui_size, ui_time, -+ ui_end) -+ self.fo.write(out) - self.fo.flush() - finally: - self._lock.release() -@@ -502,24 +578,40 @@ class TextMultiFileMeter(MultiFileMeter): - self._lock.acquire() - try: - format = "%-30.30s %6.6s %8.8s %9.9s" -- fn = meter.basename -+ fn = meter.text or meter.basename - size = meter.last_amount_read - fsize = format_number(size) + 'B' - et = meter.re.elapsed_time() -- fet = format_time(et, 1) -- frate = format_number(size / et) + 'B/s' -- -- out = '%-79.79s' % (format % (fn, fsize, fet, frate)) -- self.fo.write('\r' + out + '\n') -+ frate = format_number(et and size / et) + 'B/s' -+ df = self.finished_files -+ tf = self.numfiles or 1 -+ -+ total_size = format_number(size) -+ text = meter.text or meter.basename -+ if tf > 1: -+ text = '(%u/%u): %s' % (df, tf, text) -+ -+ tl = TerminalLine(8) -+ if tl._llen > 80: -+ use_hours = True # For big screens, make it more readable. -+ time_len = 9 -+ else: -+ use_hours = False -+ time_len = 7 -+ ui_size = tl.add(' | %5sB' % total_size) -+ ui_time = tl.add(' %*s' % (time_len, format_time(et, use_hours))) -+ ui_end, not_done = _term_add_end(tl, meter.size, size) -+ out = '\r%-*.*s%s%s%s\n' % (tl.rest(), tl.rest(), text, -+ ui_size, ui_time, ui_end) -+ self.fo.write(out) - finally: - self._lock.release() -- self._do_update_meter(meter, now) - - def _do_failure_meter(self, meter, message, now): - self._lock.acquire() - try: - format = "%-30.30s %6.6s %s" -- fn = meter.basename -+ fn = meter.text or meter.basename - if type(message) in (type(''), type(u'')): - message = message.splitlines() - if not message: message = [''] -@@ -536,15 +628,6 @@ class TextMultiFileMeter(MultiFileMeter): - pass - finally: - self._lock.release() -- -- def _do_end(self, now): -- self._do_update_meter(None, now) -- self._lock.acquire() -- try: -- self.fo.write('\n') -- self.fo.flush() -- finally: -- self._lock.release() - - ###################################################################### - # support classes and functions -@@ -658,6 +741,8 @@ def format_time(seconds, use_hours=0): - if seconds is None or seconds < 0: - if use_hours: return '--:--:--' - else: return '--:--' -+ elif seconds == float('inf'): -+ return 'Infinite' - else: - seconds = int(seconds) - minutes = seconds / 60 -@@ -722,9 +807,77 @@ def _tst(fn, cur, tot, beg, size, *args): - time.sleep(delay) - tm.end(size) - -+def _mtst(datas, *args): -+ print '-' * 79 -+ tm = TextMultiFileMeter(threaded=False) -+ -+ dl_sizes = {} -+ -+ num = 0 -+ total_size = 0 -+ dl_total_size = 0 -+ for data in datas: -+ dl_size = None -+ if len(data) == 2: -+ fn, size = data -+ dl_size = size -+ if len(data) == 3: -+ fn, size, dl_size = data -+ nm = tm.newMeter() -+ nm.start(fn, "http://www.example.com/path/to/fn/" + fn, fn, size, -+ text=fn) -+ num += 1 -+ assert dl_size is not None -+ dl_total_size += dl_size -+ dl_sizes[nm] = dl_size -+ if size is None or total_size is None: -+ total_size = None -+ else: -+ total_size += size -+ tm.start(num, total_size) -+ -+ num = 0 -+ off = 0 -+ for (inc, delay) in args: -+ off += 1 -+ while num < ((dl_total_size * off) / len(args)): -+ num += inc -+ for nm in tm.meters[:]: -+ if dl_sizes[nm] <= num: -+ nm.end(dl_sizes[nm]) -+ tm.removeMeter(nm) -+ else: -+ nm.update(num) -+ time.sleep(delay) -+ assert not tm.meters -+ - if __name__ == "__main__": - # (1/2): subversion-1.4.4-7.x86_64.rpm 2.4 MB / 85 kB/s 00:28 - # (2/2): mercurial-0.9.5-6.fc8.x86_64.rpm 924 kB / 106 kB/s 00:08 -+ if len(sys.argv) >= 2 and sys.argv[1] == 'multi': -+ _mtst((("sm-1.0.0-1.fc8.i386.rpm", 1000), -+ ("s-1.0.1-1.fc8.i386.rpm", 5000), -+ ("m-1.0.1-2.fc8.i386.rpm", 10000)), -+ (100, 0.33), (500, 0.25), (1000, 0.1)) -+ -+ _mtst((("sm-1.0.0-1.fc8.i386.rpm", 1000), -+ ("s-1.0.1-1.fc8.i386.rpm", 5000), -+ ("m-1.0.1-2.fc8.i386.rpm", None, 10000)), -+ (100, 0.33), (500, 0.25), (1000, 0.1)) -+ -+ _mtst((("sm-1.0.0-1.fc8.i386.rpm", 1000), -+ ("s-1.0.1-1.fc8.i386.rpm", 2500000), -+ ("m-1.0.1-2.fc8.i386.rpm", 10000)), -+ (10, 0.2), (50, 0.1), (1000, 0.1)) -+ -+ _mtst((("sm-1.0.0-1.fc8.i386.rpm", 1000), -+ ("s-1.0.1-1.fc8.i386.rpm", None, 2500000), -+ ("m-1.0.1-2.fc8.i386.rpm", None, 10000)), -+ (10, 0.2), (50, 0.1), (1000, 0.1)) -+ # (10, 0.2), (100, 0.1), (100, 0.1), (100, 0.25)) -+ # (10, 0.2), (100, 0.1), (100, 0.1), (100, 0.25)) -+ sys.exit(0) -+ - if len(sys.argv) >= 2 and sys.argv[1] == 'total': - text_meter_total_size(1000 + 10000 + 10000 + 1000000 + 1000000 + - 1000000 + 10000 + 10000 + 10000 + 1000000) diff --git a/python-urlgrabber/python-urlgrabber.nm b/python-urlgrabber/python-urlgrabber.nm index 1fe1083..171c836 100644 --- a/python-urlgrabber/python-urlgrabber.nm +++ b/python-urlgrabber/python-urlgrabber.nm @@ -4,8 +4,8 @@ ############################################################################### name = python-urlgrabber -version = 3.9.1 -release = 4 +version = 3.10.1 +release = 1 arch = noarch thisapp = urlgrabber-%{version} @@ -20,7 +20,7 @@ description throttling, authentication, proxies and more. end -source_dl = +source_dl = http://urlgrabber.baseurl.org/download/ build requires hooks/post-receive -- IPFire 3.x development tree

1 0

[git.ipfire.org] IPFire 3.x development tree branch, master, updated. fb50fbb2b4e08194db20f0f7bfd0ce415786b18a
by git＠ipfire.org 26 Nov '16

26 Nov '16

1 0

[git.ipfire.org] IPFire 3.x development tree branch, master, updated. 659ef6bffcd217994842073ef051a6aaf9141083
by git＠ipfire.org 26 Nov '16

26 Nov '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "IPFire 3.x development tree". The branch, master has been updated via 659ef6bffcd217994842073ef051a6aaf9141083 (commit) from c0f84b30fafdf0f5beabcdaddecb6d6795aee630 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 659ef6bffcd217994842073ef051a6aaf9141083 Author: Michael Tremer <michael.tremer(a)ipfire.org> Date: Sat Nov 26 12:39:56 2016 +0000 ddrescue: New package ddrescue is a data recovery tool Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> ----------------------------------------------------------------------- Summary of changes: ddrescue/ddrescue.nm | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 ddrescue/ddrescue.nm Difference in files: diff --git a/ddrescue/ddrescue.nm b/ddrescue/ddrescue.nm new file mode 100644 index 0000000..be34d02 --- /dev/null +++ b/ddrescue/ddrescue.nm @@ -0,0 +1,45 @@ +############################################################################### +# IPFire.org - An Open Source Firewall Solution # +# Copyright (C) - IPFire Development Team <info(a)ipfire.org> # +############################################################################### + +name = ddrescue +version = 1.21 +release = 1 + +groups = System/Tools +url = http://gnu.org/software/ddrescue +license = GPLv2+ +summary = Data recovery tool + +description + GNU ddrescue is a data recovery tool. It copies data from one file or + block device (hard disc, cdrom, etc) to another, trying to rescue the + good parts first in case of read errors. +end + +source_dl = http://ftp.gnu.org/gnu/ddrescue/ +sources = %{thisapp}.tar.lz + +build + # ddrescue comes with a non-traditional configure script + configure_options = \ + --prefix=%{prefix} \ + --enable-linux \ + CPPFLAGS="%{CPPFLAGS}" \ + CFLAGS="%{CFLAGS}" \ + LDFLAGS="%{LDFLAGS}" \ + CXXFLAGS="%{CXXFLAGS}" + + test + make check + end +end + +packages + package %{name} + + package %{name}-debuginfo + template DEBUGINFO + end +end hooks/post-receive -- IPFire 3.x development tree

1 0

[git.ipfire.org] IPFire 3.x development tree branch, master, updated. c0f84b30fafdf0f5beabcdaddecb6d6795aee630
by git＠ipfire.org 26 Nov '16

26 Nov '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "IPFire 3.x development tree". The branch, master has been updated via c0f84b30fafdf0f5beabcdaddecb6d6795aee630 (commit) via c8e80923d7c22a1786b6dc7e086ed6439abb896a (commit) from 891f6b1037278049edc8a55f62e8e8698c165a08 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit c0f84b30fafdf0f5beabcdaddecb6d6795aee630 Author: Michael Tremer <michael.tremer(a)ipfire.org> Date: Sat Nov 26 12:22:09 2016 +0000 libvirt: Make Jonatan Schlag maintainer Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> commit c8e80923d7c22a1786b6dc7e086ed6439abb896a Author: Jonatan Schlag <jonatan.schlag(a)ipfire.org> Date: Sat Nov 26 08:12:54 2016 +0000 Update libvirt to 2.4 This patch update libvirt to version 2.4 for changelog see: https://libvirt.org/news-2016.html This patch also add some new configure switches: * We build now only for qemu/kvm. The build for VMware and phyp are now disabled. * And libvirt is now explicitly built without SELinux. This change was necessary because of the fact that a build dependency of libvirt required SELinux and so libvirt is also linked against SELinux because it is available in the build environment. Until we get completely rid of SELinux we should use these switches. So we have a libvirt that does not require SELinux otherwise, we would have to release a version when we dropped SELinux completely, which would be stupid. Signed-off-by: Jonatan Schlag <jonatan.schlag(a)ipfire.org> Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> ----------------------------------------------------------------------- Summary of changes: libvirt/libvirt.nm | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) Difference in files: diff --git a/libvirt/libvirt.nm b/libvirt/libvirt.nm index fc15edf..cf398d3 100644 --- a/libvirt/libvirt.nm +++ b/libvirt/libvirt.nm @@ -4,10 +4,10 @@ ############################################################################### name = libvirt -version = 2.3.0 -release = 1 +version = 2.4.0 +release = 2 -maintainer = Ben Schweikert <ben.schweikert(a)ipfire.org> +maintainer = Jonatan Schlag <jonatan.schlag(a)ipfire.org> groups = Applications/Virtualization url = http://www.libvirt.org/ license = LGPLv2.1+ @@ -50,7 +50,9 @@ build --libexecdir=/usr/lib \ --disable-static \ --with-qemu \ + --without-vmware \ --without-esx \ + --without-phyp \ --without-hal \ --without-lxc \ --without-openvz \ @@ -64,6 +66,9 @@ build --with-storage-disk \ --with-avahi \ --with-udev \ + --without-secdriver-selinux \ + --without-selinux \ + --without-selinux-mount \ --with-libvirtd end hooks/post-receive -- IPFire 3.x development tree

1 0

[git.ipfire.org] IPFire 2.x development tree branch, next, updated. 86e9d04bfb73eb256682a567e187fe1e5cdcc3ca
by git＠ipfire.org 25 Nov '16

25 Nov '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "IPFire 2.x development tree". The branch, next has been updated via 86e9d04bfb73eb256682a567e187fe1e5cdcc3ca (commit) from bc4a68812bb131be6a8413f69909bf6a3d5a89a2 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 86e9d04bfb73eb256682a567e187fe1e5cdcc3ca Author: Michael Tremer <michael.tremer(a)ipfire.org> Date: Fri Nov 25 17:45:39 2016 +0000 unbound: Deactivate qname-minimization & harden-below-nxdomain This causes trouble when you try to resolve a record like a.b.blah.com where b.blah.com responds with NXDOMAIN. unbound won't try to resolve a.b.blah.com because it is assumed that everything longer than b.blah.com does not exist which is probably not good usability. Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> ----------------------------------------------------------------------- Summary of changes: config/unbound/unbound.conf | 2 -- 1 file changed, 2 deletions(-) Difference in files: diff --git a/config/unbound/unbound.conf b/config/unbound/unbound.conf index 3f724d8..c9b01b8 100644 --- a/config/unbound/unbound.conf +++ b/config/unbound/unbound.conf @@ -42,7 +42,6 @@ server: # Privacy Options hide-identity: yes hide-version: yes - qname-minimisation: yes minimal-responses: yes # DNSSEC @@ -56,7 +55,6 @@ server: harden-short-bufsize: no harden-large-queries: yes harden-dnssec-stripped: yes - harden-below-nxdomain: yes harden-referral-path: yes harden-algo-downgrade: no use-caps-for-id: no hooks/post-receive -- IPFire 2.x development tree

1 0

[git.ipfire.org] IPFire 3.x development tree branch, master, updated. 891f6b1037278049edc8a55f62e8e8698c165a08
by git＠ipfire.org 18 Nov '16

18 Nov '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "IPFire 3.x development tree". The branch, master has been updated via 891f6b1037278049edc8a55f62e8e8698c165a08 (commit) via f6f7f9b103f411053b00477ac85a58ff44d08759 (commit) via ec5a25262dfddb1f4158ee46983ea8a4141c2e8a (commit) via 800771e9d14087fd37162f36ac4c0b7e18ca8bf1 (commit) via 9be833268d3629c13709c88b2c2203e60809ae1a (commit) via a7e57725eb5a627414336f42d30c40db2ea87305 (commit) via f171e49b41f7dd7c3ef5a1dff1e9edbc2ab37628 (commit) from 21b0276e0d7e4439f633f0888e87fb3e691bf742 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 891f6b1037278049edc8a55f62e8e8698c165a08 Author: Michael Tremer <michael.tremer(a)ipfire.org> Date: Fri Nov 18 22:25:59 2016 +0100 net-snmp: Fix FTBFS by passing -Wformat with CFLAGS Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> commit f6f7f9b103f411053b00477ac85a58ff44d08759 Author: Michael Tremer <michael.tremer(a)ipfire.org> Date: Fri Nov 18 21:52:02 2016 +0100 perl-Net-SMTP-SSL: Update to 1.04 Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> commit ec5a25262dfddb1f4158ee46983ea8a4141c2e8a Author: Michael Tremer <michael.tremer(a)ipfire.org> Date: Fri Nov 18 21:08:32 2016 +0100 perl: Update to 5.20.3 Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> commit 800771e9d14087fd37162f36ac4c0b7e18ca8bf1 Author: Michael Tremer <michael.tremer(a)ipfire.org> Date: Fri Nov 18 03:29:49 2016 +0100 perl-DBI: Update to 1.636 Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> commit 9be833268d3629c13709c88b2c2203e60809ae1a Author: Michael Tremer <michael.tremer(a)ipfire.org> Date: Thu Nov 17 01:10:45 2016 +0100 ntp: Remove ntpstat This is an additional package that has been added to the ntp package, but now fails to build. Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> commit a7e57725eb5a627414336f42d30c40db2ea87305 Author: Michael Tremer <michael.tremer(a)ipfire.org> Date: Fri Oct 28 19:06:51 2016 +0200 libart: Drop package This package is not needed by anything any more Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> commit f171e49b41f7dd7c3ef5a1dff1e9edbc2ab37628 Author: Michael Tremer <michael.tremer(a)ipfire.org> Date: Fri Oct 28 18:55:17 2016 +0200 gupnp: Update all packages Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> ----------------------------------------------------------------------- Summary of changes: gssdp/gssdp.nm | 9 +++-- gupnp-av/gupnp-av.nm | 8 +++-- gupnp-igd/gupnp-igd.nm | 10 +++--- gupnp/gupnp.nm | 9 +++-- libart/libart.nm | 34 ------------------- .../libart_lgpl-2.3.19-upstream_fix-1.patch | 28 ---------------- net-snmp/net-snmp.nm | 4 +-- ntp/ntp.nm | 31 ++---------------- ntp/patches/ntpstat-0.2-1-clksrc.patch | 12 ------- ntp/patches/ntpstat-0.2-2-multipacket.patch | 12 ------- ntp/patches/ntpstat-0.2-3-sysvars.patch | 15 --------- ntp/patches/ntpstat-0.2-4-maxerror.patch | 38 ---------------------- ntp/patches/ntpstat-0.2-5-errorbit.patch | 32 ------------------ perl-DBI/perl-DBI.nm | 28 ++++++++++++++-- perl-Net-SMTP-SSL/perl-Net-SMTP-SSL.nm | 2 +- perl/patches/perl-5.10.0-libresolv.patch | 12 ------- perl/perl.nm | 5 +-- 17 files changed, 52 insertions(+), 237 deletions(-) delete mode 100644 libart/libart.nm delete mode 100644 libart/patches/libart_lgpl-2.3.19-upstream_fix-1.patch delete mode 100644 ntp/patches/ntpstat-0.2-1-clksrc.patch delete mode 100644 ntp/patches/ntpstat-0.2-2-multipacket.patch delete mode 100644 ntp/patches/ntpstat-0.2-3-sysvars.patch delete mode 100644 ntp/patches/ntpstat-0.2-4-maxerror.patch delete mode 100644 ntp/patches/ntpstat-0.2-5-errorbit.patch delete mode 100644 perl/patches/perl-5.10.0-libresolv.patch Difference in files: diff --git a/gssdp/gssdp.nm b/gssdp/gssdp.nm index e8e08b1..43205a5 100644 --- a/gssdp/gssdp.nm +++ b/gssdp/gssdp.nm @@ -4,8 +4,8 @@ ############################################################################### name = gssdp -ver_major = 0.14 -ver_minor = 14 +ver_major = 1.0 +ver_minor = 1 version = %{ver_major}.%{ver_minor} release = 1 @@ -18,8 +18,8 @@ description GSSDP implements resource discovery and announcement over SSDP. end -source_dl = https://download.gnome.org/sources/gssdp/%{ver_major}/ -sources = %{thisapp}.tar.xz +source_dl = https://download.gnome.org/sources/%{name}/%{ver_major}/ +sources = %{thisapp}.tar.xz build @@ -32,7 +32,6 @@ end packages package %{name} - end package %{name}-devel template DEVEL diff --git a/gupnp-av/gupnp-av.nm b/gupnp-av/gupnp-av.nm index d6f3917..1c83d7d 100644 --- a/gupnp-av/gupnp-av.nm +++ b/gupnp-av/gupnp-av.nm @@ -4,7 +4,9 @@ ############################################################################### name = gupnp-av -version = 0.5.1 +version = %{ver_maj}.%{ver_min} +ver_maj = 0.12 +ver_min = 10 release = 1 groups = System/Libraries @@ -17,7 +19,8 @@ description applications using GUPnP. end -source_dl = +source_dl = https://download.gnome.org/sources/%{name}/%{ver_maj}/ +sources = %{thisapp}.tar.xz build requires @@ -31,7 +34,6 @@ end packages package %{name} - end package %{name}-devel template DEVEL diff --git a/gupnp-igd/gupnp-igd.nm b/gupnp-igd/gupnp-igd.nm index dd0981a..d561246 100644 --- a/gupnp-igd/gupnp-igd.nm +++ b/gupnp-igd/gupnp-igd.nm @@ -4,19 +4,22 @@ ############################################################################### name = gupnp-igd -version = 0.1.3 +version = %{ver_maj}.%{ver_min} +ver_maj = 0.2 +ver_min = 4 release = 1 groups = System/Libraries url = http://www.gupnp.org license = LGPL -summary = gupnp - Internet Gateway device implementation (Microsoft). +summary = gupnp - Internet Gateway device implementation (Microsoft) description This is a library to handle Internet Gateway Device port mappings. end -source_dl = +source_dl = https://download.gnome.org/sources/%{name}/%{ver_maj}/ +sources = %{thisapp}.tar.xz build requires @@ -31,7 +34,6 @@ end packages package %{name} - end package %{name}-devel template DEVEL diff --git a/gupnp/gupnp.nm b/gupnp/gupnp.nm index 2d82384..4f51020 100644 --- a/gupnp/gupnp.nm +++ b/gupnp/gupnp.nm @@ -4,9 +4,9 @@ ############################################################################### name = gupnp -ver_major = 0.20 -ver_minor = 16 version = %{ver_major}.%{ver_minor} +ver_major = 1.0 +ver_minor = 1 release = 1 groups = System/Libraries @@ -20,8 +20,8 @@ description The GUPnP API is intended to be easy to use, efficient and flexible. end -source_dl = https://download.gnome.org/sources/gupnp/%{ver_major}/ -sources = %{thisapp}.tar.xz +source_dl = https://download.gnome.org/sources/%{name}/%{ver_major}/ +sources = %{thisapp}.tar.xz build requires @@ -38,7 +38,6 @@ end packages package %{name} - end package %{name}-devel template DEVEL diff --git a/libart/libart.nm b/libart/libart.nm deleted file mode 100644 index ec47dfd..0000000 --- a/libart/libart.nm +++ /dev/null @@ -1,34 +0,0 @@ -############################################################################### -# IPFire.org - An Open Source Firewall Solution # -# Copyright (C) - IPFire Development Team <info(a)ipfire.org> # -############################################################################### - -name = libart_gpl -version = 2.3.19 -release = 1 - -groups = System/Graphics -url = http://www.gnome.org -license = LGPLv2+ -summary = Library of graphics routines used by libgnomecanvas. - -description - Graphics routines used by the GnomeCanvas widget and some other \ - applications. libart renders vector paths and the like. -end - -source_dl = -sources = %{thisapp}.tar.bz2 - -packages - package %{name} - end - - package %{name}-devel - template DEVEL - end - - package %{name}-debuginfo - template DEBUGINFO - end -end diff --git a/libart/patches/libart_lgpl-2.3.19-upstream_fix-1.patch b/libart/patches/libart_lgpl-2.3.19-upstream_fix-1.patch deleted file mode 100644 index 68d60ed..0000000 --- a/libart/patches/libart_lgpl-2.3.19-upstream_fix-1.patch +++ /dev/null @@ -1,28 +0,0 @@ -Submitted By: Randy McMurchy <randy_at_linuxfromscratch_dot_org> -Date: 2007-04-09 -Initial Package Version: 2.3.19 -Upstream Status: In upstream CVS -Origin: Upstream -Description: Fixes an error during the installation of KDE-Libs - - -diff -Naur libart_lgpl-2.3.19.orig/art_misc.h libart_lgpl-2.3.19/art_misc.h ---- libart_lgpl-2.3.19.orig/art_misc.h 2007-01-01 18:59:22.000000000 -0500 -+++ libart_lgpl-2.3.19/art_misc.h 2007-03-20 23:04:52.000000000 -0400 -@@ -34,9 +34,15 @@ - #include <libart_lgpl/art_config.h> - #endif - -+#ifdef __cplusplus -+extern "C" { -+#endif - void *art_alloc(size_t size); - void art_free(void *ptr); - void *art_realloc(void *ptr, size_t size); -+#ifdef __cplusplus -+} -+#endif /* __cplusplus */ - - /* These aren't, strictly speaking, configuration macros, but they're - damn handy to have around, and may be worth playing with for - diff --git a/net-snmp/net-snmp.nm b/net-snmp/net-snmp.nm index 8b2a7fa..dce0719 100644 --- a/net-snmp/net-snmp.nm +++ b/net-snmp/net-snmp.nm @@ -5,7 +5,7 @@ name = net-snmp version = 5.7.2 -release = 2 +release = 3 groups = Networking/Daemons url = http://net-snmp.sourceforge.net @@ -53,7 +53,7 @@ build sctp-mib rmon-mib etherlike-mib ucd-snmp/lmsensorsMib configure_options += \ - --with-cflags="%{CFLAGS}" \ + --with-cflags="%{CFLAGS} -Wformat" \ --with-ldflags="%{LDFLAGS}" \ --with-sys-contact="root@localhost" \ --with-sys-location="Unknown" \ diff --git a/ntp/ntp.nm b/ntp/ntp.nm index 780ad01..7f8f171 100644 --- a/ntp/ntp.nm +++ b/ntp/ntp.nm @@ -6,7 +6,7 @@ name = ntp version = %{ver_major}.8 ver_major = 4.2 -release = 2 +release = 3 groups = System/Daemons url = http://www.ntp.org/ @@ -21,7 +21,6 @@ description end source_dl = http://www.eecis.udel.edu/~ntp/ntp_spool/ntp4/ntp-%{ver_major}/ -sources += ntpstat-0.2.tgz # The vendor part in the NTP pool address. VENDORZONE = %{DISTRO_SNAME} @@ -35,36 +34,17 @@ build pps-tools-devel end - prepare - %{MACRO_EXTRACT_TARBALL} - - ln -svf ../ntpstat-0.2 . - - %{MACRO_PATCHES} - + prepare_cmds # Create ntp user and group. %{create_user} end configure_options += \ --sysconfdir=%{sysconfdir}/ntp/crypto \ - --with-openssl-libdir=%{libdir} \ --enable-all-clocks \ --enable-parse-clocks \ --enable-ntp-signd=%{localstatedir}/run/ntp_signd - build - ./configure %{configure_options} - - echo "#define KEYFILE \"%{sysconfdir}/ntp/keys\"" >> ntpdate/ntpdate.h - echo "#define NTP_VAR \"%{localstatedir}/log/ntpstats/\"" >> config.h - - make ${PARALLELISMFLAGS} - - # Build ntpstat. - make -C ntpstat-0.2 - end - make_install_targets += bindir=%{sbindir} install_cmds @@ -73,13 +53,6 @@ build mv -v %{BUILDROOT}%{mandir}/man{1/sntp.1,8/sntp.8} rm -rfv %{BUILDROOT}%{mandir}/man1 - # Install ntpstat. - pushd ntpstat-0.2 - mkdir -pv %{BUILDROOT}%{bindir} - install -m 755 ntpstat %{BUILDROOT}%{bindir} - install -m 644 ntpstat.1 %{BUILDROOT}%{mandir}/man8/ntpstat.8 - popd - # Fix section numbers. sed -i 's/$\.TH[a-zA-Z ]*$[1-9]$.*$/\18\2/' \ %{BUILDROOT}%{mandir}/man8/*.8 diff --git a/ntp/patches/ntpstat-0.2-1-clksrc.patch b/ntp/patches/ntpstat-0.2-1-clksrc.patch deleted file mode 100644 index c427f1f..0000000 --- a/ntp/patches/ntpstat-0.2-1-clksrc.patch +++ /dev/null @@ -1,12 +0,0 @@ -diff -up ntp-4.2.4p7/ntpstat-0.2/ntpstat.c.ntpstat ntp-4.2.4p7/ntpstat-0.2/ntpstat.c ---- ntp-4.2.4p7/ntpstat-0.2/ntpstat.c.ntpstat 2002-06-10 08:02:12.000000000 +0200 -+++ ntp-4.2.4p7/ntpstat-0.2/ntpstat.c 2009-07-20 12:22:35.000000000 +0200 -@@ -187,7 +187,7 @@ int main (void) { - else - printf("unknown source"); - -- if (!strncmp(clksrcname[clksrc],clksrcname[6],sizeof(clksrcname[6]))) { -+ if (clksrc == 6) { - // source of sync is another NTP server so check the IP address - strncpy(buff, ntpmsg.payload, sizeof(buff)); - if ((newstr = strstr (buff, REFID))) { diff --git a/ntp/patches/ntpstat-0.2-2-multipacket.patch b/ntp/patches/ntpstat-0.2-2-multipacket.patch deleted file mode 100644 index ca21257..0000000 --- a/ntp/patches/ntpstat-0.2-2-multipacket.patch +++ /dev/null @@ -1,12 +0,0 @@ -diff -up ntp-4.2.4p7/ntpstat-0.2/ntpstat.c.ntpstat ntp-4.2.4p7/ntpstat-0.2/ntpstat.c ---- ntp-4.2.4p7/ntpstat-0.2/ntpstat.c.ntpstat 2002-06-10 08:02:12.000000000 +0200 -+++ ntp-4.2.4p7/ntpstat-0.2/ntpstat.c 2009-07-20 12:22:35.000000000 +0200 -@@ -151,7 +151,7 @@ int main (void) { - /* For the reply message to be valid, the first byte should be as sent, - and the second byte should be the same, with the response bit set */ - byte1ok = ((ntpmsg.byte1&0x3F) == B1VAL); -- byte2ok = (ntpmsg.byte2 == (B2VAL|RMASK)); -+ byte2ok = ((ntpmsg.byte2 & ~MMASK) == (B2VAL|RMASK)); - if (!(byte1ok && byte2ok)) { - fprintf (stderr,"status word is 0x%02x%02x\n", ntpmsg.byte1,ntpmsg.byte2 ); - die ("return data appears to be invalid based on status word"); diff --git a/ntp/patches/ntpstat-0.2-3-sysvars.patch b/ntp/patches/ntpstat-0.2-3-sysvars.patch deleted file mode 100644 index 3f641a9..0000000 --- a/ntp/patches/ntpstat-0.2-3-sysvars.patch +++ /dev/null @@ -1,15 +0,0 @@ -diff -up ntp-4.2.6p1/ntpstat-0.2/ntpstat.c.sysvars ntp-4.2.6p1/ntpstat-0.2/ntpstat.c ---- ntp-4.2.6p1/ntpstat-0.2/ntpstat.c.sysvars 2010-05-03 11:27:47.000000000 +0200 -+++ ntp-4.2.6p1/ntpstat-0.2/ntpstat.c 2010-05-03 11:32:56.000000000 +0200 -@@ -89,9 +89,9 @@ int main (void) { - "modem"}; /* 9 */ - char *newstr; - char *dispstr; -- const char DISP[] = "rootdispersion="; -+ const char DISP[] = "rootdisp="; - const char STRATUM[] = "stratum="; -- const char POLL[] = "poll="; -+ const char POLL[] = "tc="; - const char REFID[] = "refid="; - - /* initialise timeout value */ diff --git a/ntp/patches/ntpstat-0.2-4-maxerror.patch b/ntp/patches/ntpstat-0.2-4-maxerror.patch deleted file mode 100644 index f8ab750..0000000 --- a/ntp/patches/ntpstat-0.2-4-maxerror.patch +++ /dev/null @@ -1,38 +0,0 @@ -diff -up ntp-4.2.6p1/ntpstat-0.2/ntpstat.c.maxerror ntp-4.2.6p1/ntpstat-0.2/ntpstat.c ---- ntp-4.2.6p1/ntpstat-0.2/ntpstat.c.maxerror 2010-05-03 11:37:49.000000000 +0200 -+++ ntp-4.2.6p1/ntpstat-0.2/ntpstat.c 2010-05-03 12:20:08.000000000 +0200 -@@ -89,7 +89,9 @@ int main (void) { - "modem"}; /* 9 */ - char *newstr; - char *dispstr; -+ char *delaystr; - const char DISP[] = "rootdisp="; -+ const char DELAY[] = "rootdelay="; - const char STRATUM[] = "stratum="; - const char POLL[] = "tc="; - const char REFID[] = "refid="; -@@ -235,16 +237,19 @@ int main (void) { - /* Set the position of the start of the string to - "rootdispersion=" part of the string. */ - strncpy(buff, ntpmsg.payload, sizeof(buff)); -- if ((newstr = strstr (buff, DISP))) { -- newstr += sizeof(DISP) - 1; -- dispstr = strtok(newstr,"."); -+ if ((dispstr = strstr (buff, DISP)) && (delaystr = strstr (buff, DELAY))) { -+ dispstr += sizeof(DISP) - 1; -+ dispstr = strtok(dispstr,","); -+ delaystr += sizeof(DELAY) - 1; -+ delaystr = strtok(delaystr,","); - - /* Check the resultant string is of a reasonable length */ -- if ((strlen (dispstr) == 0) || (strlen (dispstr) > 4)) { -+ if ((strlen (dispstr) == 0) || (strlen (dispstr) > 10) || -+ (strlen (delaystr) == 0) || (strlen (delaystr) > 10)) { - printf ("accuracy unreadable\n"); - } - else { -- printf(" time correct to within %s ms\n",dispstr); -+ printf(" time correct to within %.0f ms\n", atof(dispstr) + atof(delaystr) / 2.0); - } - } else { - rc=1; diff --git a/ntp/patches/ntpstat-0.2-5-errorbit.patch b/ntp/patches/ntpstat-0.2-5-errorbit.patch deleted file mode 100644 index 06e9342..0000000 --- a/ntp/patches/ntpstat-0.2-5-errorbit.patch +++ /dev/null @@ -1,32 +0,0 @@ -diff -up ntp-4.2.6p4/ntpstat-0.2/ntpstat.c.errorbit ntp-4.2.6p4/ntpstat-0.2/ntpstat.c ---- ntp-4.2.6p4/ntpstat-0.2/ntpstat.c.errorbit 2011-10-06 13:41:38.591669772 +0200 -+++ ntp-4.2.6p4/ntpstat-0.2/ntpstat.c 2011-10-06 16:50:01.708315811 +0200 -@@ -104,6 +104,7 @@ int main (void) { - FD_ZERO(&fds); - - inet_aton("127.0.0.1", &address); -+ memset(&sock, 0, sizeof (sock));; - sock.sin_family = AF_INET; - sock.sin_addr = address; - sock.sin_port = htons(NTP_PORT); -@@ -159,15 +160,18 @@ int main (void) { - die ("return data appears to be invalid based on status word"); - } - -- if (!(ntpmsg.byte2 | EMASK)) { -+ if (ntpmsg.byte2 & EMASK) { - fprintf (stderr,"status byte2 is %02x\n", ntpmsg.byte2 ); - die ("error bit is set in reply"); - } - -- if (!(ntpmsg.byte2 | MMASK)) { -+ /* ignore the more bit */ -+#if 0 -+ if (ntpmsg.byte2 & MMASK) { - fprintf (stderr,"status byte2 is %02x\n", ntpmsg.byte2 ); - fprintf (stderr,"More bit unexpected in reply"); - } -+#endif - - /* if the leap indicator (LI), which is the two most significant bits - in status byte1, are both one, then the clock is not synchronised. */ diff --git a/perl-DBI/perl-DBI.nm b/perl-DBI/perl-DBI.nm index 8bc06bc..546c034 100644 --- a/perl-DBI/perl-DBI.nm +++ b/perl-DBI/perl-DBI.nm @@ -4,8 +4,8 @@ ############################################################################### name = perl-DBI -version = 1.633 -release = 1.1 +version = 1.636 +release = 1 thisapp = DBI-%{version} groups = Development/Libraries @@ -22,10 +22,31 @@ source_dl = http://search.cpan.org/CPAN/authors/id/T/TI/TIMB/ build requires + perl(Carp) + perl(Cwd) + perl(Data::Dumper) + perl(DynaLoader) + perl(Errno) + perl(Exporter) perl(ExtUtils::MakeMaker) perl(ExtUtils::ParseXS) - perl(Carp) + perl(File::Basename) + perl(File::Find) + perl(File::Spec) + perl(Getopt::Long) + perl(IO::Dir) + perl(IO::File) + perl(IO::Select) + perl(IPC::Open3) + perl(Math::BigInt) + perl(Scalar::Util) + perl(Storable) + perl(Symbol) + perl(Tie::Hash) perl(Test::Simple) + perl(base) + perl(strict) + perl-devel pakfire >= 0.9.26-3.1 end @@ -51,6 +72,7 @@ packages requires perl(:MODULE_COMPAT_%{perl_version}) + perl(Math::BigInt) end end diff --git a/perl-Net-SMTP-SSL/perl-Net-SMTP-SSL.nm b/perl-Net-SMTP-SSL/perl-Net-SMTP-SSL.nm index f6ea363..47b7224 100644 --- a/perl-Net-SMTP-SSL/perl-Net-SMTP-SSL.nm +++ b/perl-Net-SMTP-SSL/perl-Net-SMTP-SSL.nm @@ -4,7 +4,7 @@ ############################################################################### name = perl-Net-SMTP-SSL -version = 1.03 +version = 1.04 release = 1 arch = noarch thisapp = Net-SMTP-SSL-%{version} diff --git a/perl/patches/perl-5.10.0-libresolv.patch b/perl/patches/perl-5.10.0-libresolv.patch deleted file mode 100644 index 66b6a68..0000000 --- a/perl/patches/perl-5.10.0-libresolv.patch +++ /dev/null @@ -1,12 +0,0 @@ -diff -up perl-5.10.0/Configure.didi perl-5.10.0/Configure ---- perl-5.10.0/Configure.didi 2007-12-18 11:47:07.000000000 +0100 -+++ perl-5.10.0/Configure 2008-07-21 10:51:16.000000000 +0200 -@@ -1327,7 +1327,7 @@ libswanted_uselargefiles='' - : set usesocks on the Configure command line to enable socks. - : List of libraries we want. - : If anyone needs extra -lxxx, put those in a hint file. --libswanted="socket bind inet nsl nm ndbm gdbm dbm db malloc dl ld sun" -+libswanted="socket resolv inet nsl nm ndbm gdbm dbm db malloc dl ld sun" - libswanted="$libswanted m crypt sec util c cposix posix ucb bsd BSD" - : We probably want to search /usr/shlib before most other libraries. - : This is only used by the lib/ExtUtils/MakeMaker.pm routine extliblist. diff --git a/perl/perl.nm b/perl/perl.nm index 62f49fb..71537a2 100644 --- a/perl/perl.nm +++ b/perl/perl.nm @@ -4,9 +4,9 @@ ############################################################################### name = perl -version = 5.20.2 +version = 5.20.3 # Never reset release in this package, just increase. -release = 14 +release = 15 perl_epoch = 2 thisver = %{perl_epoch}:%{version}-%{_release} @@ -150,6 +150,7 @@ packages provides # Compat provides + perl(:MODULE_COMPAT_5.20.3) perl(:MODULE_COMPAT_5.20.2) perl(:MODULE_COMPAT_5.20.1) perl(:MODULE_COMPAT_5.20.0) hooks/post-receive -- IPFire 3.x development tree

1 0

[git.ipfire.org] IPFire 2.x development tree branch, next, updated. bc4a68812bb131be6a8413f69909bf6a3d5a89a2
by git＠ipfire.org 17 Nov '16

17 Nov '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "IPFire 2.x development tree". The branch, next has been updated via bc4a68812bb131be6a8413f69909bf6a3d5a89a2 (commit) from c6bc0fb03e0dbb9f1ba34d42195d0601b55891c1 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit bc4a68812bb131be6a8413f69909bf6a3d5a89a2 Author: Alexander Marx <alexander.marx(a)ipfire.org> Date: Mon Oct 31 12:19:15 2016 +0100 BUG11242: Fix for adding 2 VPN Hosts/network with same name If one has an IPSec network named "aaa" and an OpenVPn Host with the same name it was not possible to group them together because of the same name. Now the Network type is also checked wich allows Entries with same name, but different networks. Fixes: #11242 Signed-off-by: Alexander Marx <alexander.marx(a)ipfire.org> Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> ----------------------------------------------------------------------- Summary of changes: html/cgi-bin/fwhosts.cgi | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) Difference in files: diff --git a/html/cgi-bin/fwhosts.cgi b/html/cgi-bin/fwhosts.cgi index 35afad3..1b0fe07 100644 --- a/html/cgi-bin/fwhosts.cgi +++ b/html/cgi-bin/fwhosts.cgi @@ -624,9 +624,9 @@ if ($fwhostsettings{'ACTION'} eq 'savegrp') } #check if host/net exists in grp - my $test="$grp,$fwhostsettings{'oldremark'},@target"; + my $test="$grp,$fwhostsettings{'oldremark'},@target,$type"; foreach my $key (keys %customgrp) { - my $test1="$customgrp{$key}[0],$customgrp{$key}[1],$customgrp{$key}[2]"; + my $test1="$customgrp{$key}[0],$customgrp{$key}[1],$customgrp{$key}[2],$customgrp{$key}[3]"; if ($test1 eq $test){ $errormessage=$Lang::tr{'fwhost err isingrp'}; $fwhostsettings{'update'} = 'on'; hooks/post-receive -- IPFire 2.x development tree

1 0

[git.ipfire.org] IPFire 3.x development tree branch, master, updated. 21b0276e0d7e4439f633f0888e87fb3e691bf742
by git＠ipfire.org 16 Nov '16

16 Nov '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "IPFire 3.x development tree". The branch, master has been updated via 21b0276e0d7e4439f633f0888e87fb3e691bf742 (commit) from e4770a6b5ec5c85e0d50baddb9fb8c163f0bafee (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 21b0276e0d7e4439f633f0888e87fb3e691bf742 Author: Michael Tremer <michael.tremer(a)ipfire.org> Date: Wed Nov 16 21:37:46 2016 +0000 setup: Configure swappiness This will turn off any preemptive swapping and will only allow swapping to avoid any OOM conditions. This will help to keep the system responsive, avoids latency and saves writing on storage. Signed-off-by: Michael Tremer <michael.tremer(a)ipfire.org> ----------------------------------------------------------------------- Summary of changes: setup/setup.nm | 10 +++++++--- setup/sysctl/swappiness.conf | 5 +++++ 2 files changed, 12 insertions(+), 3 deletions(-) create mode 100644 setup/sysctl/swappiness.conf Difference in files: diff --git a/setup/setup.nm b/setup/setup.nm index 61d3ba6..75c9da2 100644 --- a/setup/setup.nm +++ b/setup/setup.nm @@ -5,7 +5,7 @@ name = setup version = 3.0 -release = 7 +release = 8 arch = noarch groups = Base Build System/Base @@ -45,8 +45,12 @@ build cp -vf %{DIR_APP}/${i} %{BUILDROOT}/etc || exit 1; \ done - touch %{BUILDROOT}/etc/{environment,fstab} - chmod -v 0400 %{BUILDROOT}/etc/{,g}shadow + touch %{BUILDROOT}%{sysconfdir}/{environment,fstab} + chmod -v 0400 %{BUILDROOT}%{sysconfdir}/{,g}shadow + + mkdir -pv %{BUILDROOT}%{sysconfdir}/sysctl.d + install -m 644 %{DIR_APP}/sysctl/swappiness.conf \ + %{BUILDROOT}%{sysconfdir}/sysctl.d/swappiness.conf end end diff --git a/setup/sysctl/swappiness.conf b/setup/sysctl/swappiness.conf new file mode 100644 index 0000000..253b75c --- /dev/null +++ b/setup/sysctl/swappiness.conf @@ -0,0 +1,5 @@ +# This will turn off any preemptive swapping and will only allow swapping to +# avoid any OOM conditions. This will help to keep the system responsive, +# avoids latency and saves writing on storage. + +vm.swappiness = 0 hooks/post-receive -- IPFire 3.x development tree

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

IPFire-SCM November 2016