This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "IPFire 2.x development tree".
The branch, next has been updated via 9fda5e9258de75030316a046bf40f0e2779d69e6 (commit) via e04f86b4490c2f22c93fd1fdf9ae5d06971b8581 (commit) via e331e22bda26e53cd71d04e43d4d8a9d9306b51e (commit) via 329631dee9f0adfb512f48aac61bbfdfe8f09a7a (commit) via 645da4f33dd81df39991b510cca22dbead5d5255 (commit) via 39c0b5b3bcd9d4cf57437d1988df834a1800c66c (commit) via e7bf3de34b8e382b229338377d9f20342bdc6bf6 (commit) via 9d79aea24b8554172464600252814b0765434e47 (commit) via d093117fe4035373f6a921d8cab6ef39f652d7fe (commit) via 74713741e617acc6d5eb7a74da1188a437a6413c (commit) via 375c4480d75fc0802cdaa3ac68712348ab8448c1 (commit) via ff9fdd63889fc39bb64ff5d38e66a5e4fbd119e6 (commit) via 15f7b86597c816426f5b4a9e728e257fb29969bf (commit) via d894ef3615b1f3d2a4ed8f78c8fb24083dc9cc57 (commit) from bc6257f49045c91a36a96e2ead87b52e6c7bfd8a (commit)
Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below.
- Log ----------------------------------------------------------------- commit 9fda5e9258de75030316a046bf40f0e2779d69e6 Author: Michael Tremer michael.tremer@ipfire.org Date: Sun Jan 7 19:51:07 2018 +0000
core118: Ship update accelerator downloader
Signed-off-by: Michael Tremer michael.tremer@ipfire.org
commit e04f86b4490c2f22c93fd1fdf9ae5d06971b8581 Author: Justin Luth justin_luth@sil.org Date: Sat Dec 30 09:25:50 2017 +0300
Fix bug 11567 updxlrator: don't prematurely release lock file
With Microsoft's new style of downloading updates, where portions of a patch are requested multiple times per second, it has become extremely common for downloads to reach > 100%. Due to an early unlinking of the "lock" file, there is a big window of opportunity (between the unlink and wget actually saving some data) for multiple download/wget threads to start, adding to the same file. So not only is bandwidth wasted by duplicate downloads running simultaneously, but the resulting file is corrupt anyway.
The problem is noticed more often by low bandwidth users (who need the benefits of updxlrator the most) because then wget's latency is even longer, creating a very wide window of opportunity.
Ultimately, this needs something like "flock", where the file is set and tested in one operation. But for now, settle with the current test / create lock solution, and just stop unnecessarily releasing the lock.
Since the file already exists as a lock when wget starts, wget now must ALWAYS run with --continue, which works fine on a zero-sized file.
Signed-off-by: Justin Luth jluth@mail.com Signed-off-by: Michael Tremer michael.tremer@ipfire.org
commit e331e22bda26e53cd71d04e43d4d8a9d9306b51e Author: Justin Luth justin_luth@sil.org Date: Sat Dec 30 08:48:37 2017 +0300
updxlrator: show hostaddr in debuglog
There is nowhere in the debuglog any indication of which client is requesting the file that updxlrator is providing (or caching). Especially for those huge Windows 10 downloads, it is valuable to see which client is requesting them, especially when the same client requests the same download multiple times a second.
This only impacts users who turn on debugging.
Signed-off-by: Justin Luth jluth@mail.com Signed-off-by: Michael Tremer michael.tremer@ipfire.org
commit 329631dee9f0adfb512f48aac61bbfdfe8f09a7a Author: Justin Luth jluth@mail.com Date: Sat Dec 30 22:12:01 2017 +0300
Fix bug 11558 updxlrator: use mirror mode for SHA1, filenames
Most Microsoft updates now contain an SHA1 hash in the filename. Since these files are uniquely identifiable, use mirror mode (which creates a hash of just the filename instead of the entire URL) to cache them. (But first check the URL cache to see if it has been downloaded as a URL already.)
This is a HUGELY needed fix. Windows 10 updates are 5+ GB per month, and we lose several days of bandwidth downloading duplicates from different mirrors. Sometimes a single client will request the same patch from multiple mirrors. That's bad. This patch will save a ton of bandwidth, and lots of disk space.
The patch limits the SHA1 test to microsoft only, but it could be easily expanded to other vendors if there is a need.
Signed-off-by: Justin Luth jluth@mail.com Signed-off-by: Michael Tremer michael.tremer@ipfire.org
commit 645da4f33dd81df39991b510cca22dbead5d5255 Author: Michael Tremer michael.tremer@ipfire.org Date: Sun Jan 7 19:28:28 2018 +0000
core118: Ship updated update accelerator
Signed-off-by: Michael Tremer michael.tremer@ipfire.org
commit 39c0b5b3bcd9d4cf57437d1988df834a1800c66c Author: Justin Luth jluth@mail.com Date: Fri Dec 29 17:12:27 2017 +0300
Fix bug 10504: match download's sourceurl mangling in, updxlrator
Updatexlrator stores its files in a hash of the URL.
The download utility mangles the URL for [+/~], but the updxlrator only does it for [/]. Thus, download stores the result as one hash, and updxlrator looks for it with a different hash. The result is that the file is re-downloaded every time by both the client, and updxlrator.
This is fixed by making updxlrator mangle the url in the same way as the downloader. apt-get install g++ would be a good test for this.
Signed-off-by: Justin Luth jluth@mail.com Signed-off-by: Michael Tremer michael.tremer@ipfire.org
commit e7bf3de34b8e382b229338377d9f20342bdc6bf6 Author: Michael Tremer michael.tremer@ipfire.org Date: Sun Jan 7 19:22:27 2018 +0000
core118: Ship updated ids.cgi
Signed-off-by: Michael Tremer michael.tremer@ipfire.org
commit 9d79aea24b8554172464600252814b0765434e47 Author: Matthias Fischer matthias.fischer@ipfire.org Date: Sat Jan 6 09:18:39 2018 +0100
snort 2.9.11.1: 'ids.cgi' - Update for snort rules download url
Signed-off-by: Matthias Fischer matthias.fischer@ipfire.org Signed-off-by: Michael Tremer michael.tremer@ipfire.org
commit d093117fe4035373f6a921d8cab6ef39f652d7fe Author: Michael Tremer michael.tremer@ipfire.org Date: Sun Jan 7 19:21:35 2018 +0000
core118: Ship updated snort
Signed-off-by: Michael Tremer michael.tremer@ipfire.org
commit 74713741e617acc6d5eb7a74da1188a437a6413c Author: Matthias Fischer matthias.fischer@ipfire.org Date: Fri Jan 5 18:28:00 2018 +0100
snort: Update to 2.9.11.1
For details see:
Release notes: https://snort.org/downloads/snort/release_notes_2.9.11.1.txt
Changelog: https://snort.org/downloads/snort/changelog_2.9.11.1.txt
Best, Matthias
Signed-off-by: Matthias Fischer matthias.fischer@ipfire.org Signed-off-by: Michael Tremer michael.tremer@ipfire.org
commit 375c4480d75fc0802cdaa3ac68712348ab8448c1 Author: Michael Tremer michael.tremer@ipfire.org Date: Sun Jan 7 19:19:12 2018 +0000
core118: Ship updated fireinfo.cgi
Signed-off-by: Michael Tremer michael.tremer@ipfire.org
commit ff9fdd63889fc39bb64ff5d38e66a5e4fbd119e6 Author: Peter Müller peter.mueller@link38.eu Date: Sun Jan 7 11:01:36 2018 +0100
use HTTPS for links to fireinfo.ipfire.org
Since fireinfo.ipfire.org is now supporting HTTPS, the links in the WebUI should point to the secure version of the site.
Signed-off-by: Peter Müller peter.mueller@link38.eu Signed-off-by: Michael Tremer michael.tremer@ipfire.org
commit 15f7b86597c816426f5b4a9e728e257fb29969bf Author: Peter Müller peter.mueller@link38.eu Date: Sun Jan 7 13:55:05 2018 +0100
ship updated showrequestfromcountry.cgi file
Signed-off-by: Peter Müller peter.mueller@link38.eu Signed-off-by: Michael Tremer michael.tremer@ipfire.org
commit d894ef3615b1f3d2a4ed8f78c8fb24083dc9cc57 Author: Peter Müller peter.mueller@link38.eu Date: Sun Jan 7 13:52:11 2018 +0100
fix GeoIP lookup in showrequestfromcountry.dat
This issue was caused by the rewrite of the perl GeoIP library.
Fixes #11571.
Signed-off-by: Peter Müller peter.mueller@link38.eu Tested-by: Matthias Fischer matthias.fischer@ipfire.org Signed-off-by: Michael Tremer michael.tremer@ipfire.org
-----------------------------------------------------------------------
Summary of changes: config/rootfiles/core/118/filelists/files | 5 +++++ .../{oldcore/104 => core/118}/filelists/snort | 0 config/rootfiles/core/118/update.sh | 6 ++++-- config/updxlrator/download | 12 ++++++------ config/updxlrator/updxlrator | 19 +++++++++++++++++-- html/cgi-bin/fireinfo.cgi | 4 ++-- html/cgi-bin/ids.cgi | 4 ++-- html/cgi-bin/logs.cgi/showrequestfromcountry.dat | 2 +- lfs/snort | 4 ++-- 9 files changed, 39 insertions(+), 17 deletions(-) copy config/rootfiles/{oldcore/104 => core/118}/filelists/snort (100%)
Difference in files: diff --git a/config/rootfiles/core/118/filelists/files b/config/rootfiles/core/118/filelists/files index cbc59893f..6780a011b 100644 --- a/config/rootfiles/core/118/filelists/files +++ b/config/rootfiles/core/118/filelists/files @@ -4,4 +4,9 @@ etc/httpd/conf/loadmodule.conf etc/httpd/conf/vhosts.d/captive.conf etc/httpd/conf/vhosts.d/ipfire-interface-ssl.conf etc/httpd/conf/vhosts.d/ipfire-interface.conf +srv/web/ipfire/cgi-bin/fireinfo.cgi +srv/web/ipfire/cgi-bin/ids.cgi +srv/web/ipfire/cgi-bin/logs.cgi/showrequestfromcountry.dat +usr/sbin/updxlrator var/ipfire/langs +var/ipfire/updatexlrator/bin/download diff --git a/config/rootfiles/core/118/filelists/snort b/config/rootfiles/core/118/filelists/snort new file mode 120000 index 000000000..9406ce01c --- /dev/null +++ b/config/rootfiles/core/118/filelists/snort @@ -0,0 +1 @@ +../../../common/snort \ No newline at end of file diff --git a/config/rootfiles/core/118/update.sh b/config/rootfiles/core/118/update.sh index 18a167c3b..60173609f 100644 --- a/config/rootfiles/core/118/update.sh +++ b/config/rootfiles/core/118/update.sh @@ -31,6 +31,9 @@ for (( i=1; i<=$core; i++ )); do rm -f /var/cache/pakfire/core-upgrade-*-$i.ipfire done
+# Stop services +/etc/init.d/snort stop + # Delete files rm -rvf \ /etc/httpd/conf.d/php5.conf \ @@ -42,8 +45,6 @@ rm -rvf \ /usr/lib/apache/libphp5.so \ /usr/lib/php
-# Stop services - # Extract files extract_files
@@ -55,6 +56,7 @@ ldconfig
# Start services /etc/init.d/apache restart +/etc/init.d/snort start
# This update need a reboot... #touch /var/run/need_reboot diff --git a/config/updxlrator/download b/config/updxlrator/download index dbc722c23..afa6e6cb9 100644 --- a/config/updxlrator/download +++ b/config/updxlrator/download @@ -30,7 +30,6 @@ my $unique=0; my $mirror=1;
my %dlinfo=(); -my $wgetContinueFlag="";
my $vendorid = $ARGV[0]; if (!defined($vendorid) || $vendorid eq '') { exit; } my $sourceurl = $ARGV[1]; if (!defined($sourceurl) || $sourceurl eq '') { exit; } @@ -57,16 +56,15 @@ if($restartdl == 0) # this is a new download exit if (-e "$repository/download/$vendorid/$updatefile");
- # dotzball: Why is this necessary? + # hinder multiple downloads from starting simultaneously. Create empty "lock" file. + # TODO: Another thread may sneak in between these two commands - so not fool-proof, but good enough? system("touch $repository/download/$vendorid/$updatefile"); - $wgetContinueFlag = "-nc";
} else { # this is a restart of a previous (unfinished) download # -> continue download - $wgetContinueFlag = "-c"; &writelog("Continue download: $updatefile"); }
@@ -133,7 +131,9 @@ unless($restartdl) { # this is a new download # -> download from scratch - unlink "$repository/download/$vendorid/$updatefile"; + + #already exited earlier if the file existed, and afterwards created this empty "lock", so if not empty now, another thread is already downloading it. + exit if ( -s "$repository/download/$vendorid/$updatefile" ); unlink "$repository/download/$vendorid/$updatefile.info"; }
@@ -147,7 +147,7 @@ $dlinfo{'REMOTESIZE'} = $remote_size; $dlinfo{'STATUS'} = "1"; &UPDXLT::writehash("$repository/download/$vendorid/$updatefile.info", %dlinfo);
-my $cmd = "$UPDXLT::wget $login $dlrate --user-agent="$UPDXLT::useragent" -q -P $repository/download/$vendorid $wgetContinueFlag $sourceurl"; +my $cmd = "$UPDXLT::wget $login $dlrate --user-agent="$UPDXLT::useragent" -q -P $repository/download/$vendorid --continue $sourceurl";
$_ = system("$cmd"); $ENV{'http_proxy'} = ''; diff --git a/config/updxlrator/updxlrator b/config/updxlrator/updxlrator index 2ddc6d8e4..ff23b3a95 100644 --- a/config/updxlrator/updxlrator +++ b/config/updxlrator/updxlrator @@ -86,6 +86,8 @@ while (<>) { && ($source_url !~ m@&@) ) { + # NOTE: check_cache will change to $mirror instead of $unique if the filename contains an SHA1 hash + # and the URL is not found in cache! $xlrator_url = &check_cache($source_url,$hostaddr,$username,"Microsoft",$unique); }
@@ -345,7 +347,9 @@ sub check_cache my $sourceurl=$_[0]; my $cfmirror=$_[4];
+ $sourceurl =~ s@%2b@+@ig; $sourceurl =~ s@%2f@/@ig; + $sourceurl =~ s@%7e@~@ig; $updfile = substr($sourceurl,rindex($sourceurl,"/")+1); $updfile =~ s@%20@ @ig;
@@ -395,10 +399,21 @@ sub check_cache ) { &debuglog("File exists in cache and is up to date"); - &debuglog("Retrieving file from cache ($updsource)"); + &debuglog("Retrieving file from cache ($updsource) for $hostaddr"); &setcachestatus("$updcachedir/$vendorid/$uuid/access.log",time); $cacheurl="http://$netsettings%7B%27GREEN_ADDRESS%27%7D:$http_port/updatecache/$vendori..."; } + elsif ( + ($cfmirror == $unique) && + ($vendorid == "microsoft") && + ($source_url =~ m@.*[0-9a-f]{40}.[^.]+@i) + ) + { + # Most Microsoft updates now have an SHA1 hash in the name. These should be treated as unique files. + # Since it wasn't found in the URL cache, switch to mirror mode and try again using just the filename. + &debuglog("SHA1: $vendorid $uuid not cached. Reprocessing as mirror $sourceurl"); + $cacheurl = &check_cache($source_url,$hostaddr,$username,$vendorid,$mirror); + } else { if (-e "$updcachedir/$vendorid/$uuid/$updfile") @@ -412,7 +427,7 @@ sub check_cache &debuglog("Free disk space: " . &diskfree($updcachedir)); &debuglog("Disk usage: " . &diskusage($updcachedir) . "% (max. $maxusage%)"); if (-e "$updcachedir/download/$vendorid/$updfile") { &debuglog("File download/$vendorid/$updfile exists"); } - &debuglog("Retrieving file from source ($updsource)"); + &debuglog("Retrieving file from source ($updsource) for $hostaddr"); if ((!$passive_mode) && (&diskusage($updcachedir) <= $maxusage) && ($remote_size <= &diskfree($updcachedir)) && (!-e "$updcachedir/download/$vendorid/$updfile")) { &debuglog("Running command $nice$apphome/bin/download $vendorid $sourceurl $cfmirror &"); diff --git a/html/cgi-bin/fireinfo.cgi b/html/cgi-bin/fireinfo.cgi index 0060a117c..08fee3d8a 100644 --- a/html/cgi-bin/fireinfo.cgi +++ b/html/cgi-bin/fireinfo.cgi @@ -67,7 +67,7 @@ if ($fireinfosettings{'ENABLE_FIREINFO'} ne "on") { <font color="$Header::colourred"> <p> $Lang::tr{'fireinfo why descr1'} - $Lang::tr{'fireinfo why descr2'}<a href="http://fireinfo.ipfire.org" target="_blank">$Lang::tr{'fireinfo why read more'}</a> + $Lang::tr{'fireinfo why descr2'}<a href="https://fireinfo.ipfire.org" target="_blank">$Lang::tr{'fireinfo why read more'}</a> </p> </font> END @@ -122,7 +122,7 @@ print <<END; <tr> <td>$Lang::tr{'fireinfo your profile id'}:</td> <td> - <a href="http://fireinfo.ipfire.org/profile/$profile_id" target="_blank">$profile_id</a> + <a href="https://fireinfo.ipfire.org/profile/$profile_id" target="_blank">$profile_id</a> </td> </tr> <tr> diff --git a/html/cgi-bin/ids.cgi b/html/cgi-bin/ids.cgi index ff48333b7..17cb23292 100644 --- a/html/cgi-bin/ids.cgi +++ b/html/cgi-bin/ids.cgi @@ -259,9 +259,9 @@ if ($snortsettings{'OINKCODE'} ne "") {
if (!$errormessage) { if ($snortsettings{'RULES'} eq 'subscripted') { - $url=" https://www.snort.org/rules/snortrules-snapshot-29110.tar.gz?oinkcode=$snort..."; + $url=" https://www.snort.org/rules/snortrules-snapshot-29111.tar.gz?oinkcode=$snort..."; } elsif ($snortsettings{'RULES'} eq 'registered') { - $url=" https://www.snort.org/rules/snortrules-snapshot-29110.tar.gz?oinkcode=$snort..."; + $url=" https://www.snort.org/rules/snortrules-snapshot-29111.tar.gz?oinkcode=$snort..."; } elsif ($snortsettings{'RULES'} eq 'community') { $url=" https://www.snort.org/rules/community"; } else { diff --git a/html/cgi-bin/logs.cgi/showrequestfromcountry.dat b/html/cgi-bin/logs.cgi/showrequestfromcountry.dat index 605873ac0..69835370b 100644 --- a/html/cgi-bin/logs.cgi/showrequestfromcountry.dat +++ b/html/cgi-bin/logs.cgi/showrequestfromcountry.dat @@ -349,7 +349,7 @@ foreach $_ (@slice) if($iface eq $country || $srcaddr ne '') { my $ccode=''; if($iface ne $country) { - $ccode = $gi->country_code_by_name($srcaddr); + $ccode = &GeoIP::lookup($srcaddr); } if($iface eq $country || $ccode eq uc($country)) { my $chain = ''; diff --git a/lfs/snort b/lfs/snort index f66bf0127..a5d0535b0 100644 --- a/lfs/snort +++ b/lfs/snort @@ -24,7 +24,7 @@
include Config
-VER = 2.9.11 +VER = 2.9.11.1
THISAPP = snort-$(VER) DL_FILE = $(THISAPP).tar.gz @@ -40,7 +40,7 @@ objects = $(DL_FILE)
$(DL_FILE) = $(DL_FROM)/$(DL_FILE)
-$(DL_FILE)_MD5 = 4f3ee0c2ad330e487273016efd0c9d64 +$(DL_FILE)_MD5 = 378e3938b2b5c8e358f942d0ffce18cc
install : $(TARGET)
hooks/post-receive -- IPFire 2.x development tree