Development December 2017

development@lists.ipfire.org

5 participants
28 discussions

Planning out Core Update 117
by Michael Tremer 12 May '18

12 May '18

Hello, so the big Core Update 115 has been shipped last week. Core Update 116 is coming today. So the next one should be a little bit calmer please :) I do not have anything so far except some smaller bug fixes here and there. Does anyone have anything big? It doesn't have to be another big update. We could also just improve IPFire where ever it is needed with loads of smaller changes and start working on 3 and other sub-projects that are floating around... Best, -Michael

5 31

[PATCH v2] show RNGD status at WebUI service page, too
by Peter Müller 27 Mar '18

27 Mar '18

Also show the status of the hardware random number generator ("RNGD") at the services.cgi page in the WebUI. Before, this was only shown at the entropy.cgi page, which was a bit inconsistent. Signed-off-by: Peter Müller <peter.mueller(a)link38.eu> --- html/cgi-bin/services.cgi | 4 +++- src/initscripts/system/rngd | 6 ++++++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/html/cgi-bin/services.cgi b/html/cgi-bin/services.cgi index 64fdbba05..eaaa5298a 100644 --- a/html/cgi-bin/services.cgi +++ b/html/cgi-bin/services.cgi @@ -56,6 +56,7 @@ my %servicenames =( $Lang::tr{'secure shell server'} => 'sshd', $Lang::tr{'vpn'} => 'charon', $Lang::tr{'web proxy'} => 'squid', + $Lang::tr{'random number generator daemon'} => 'rngd', 'OpenVPN' => 'openvpn' ); @@ -74,7 +75,8 @@ my %link =( "$Lang::tr{'intrusion detection system'} (GREEN)" => "<a href=\'ids.cgi\'>$Lang::tr{'intrusion detection system'} (GREEN)</a>", "$Lang::tr{'intrusion detection system'} (RED)" => "<a href=\'ids.cgi\'>$Lang::tr{'intrusion detection system'} (RED)</a>", "$Lang::tr{'intrusion detection system'} (ORANGE)" => "<a href=\'ids.cgi\'>$Lang::tr{'intrusion detection system'} (ORANGE)</a>", - "$Lang::tr{'intrusion detection system'} (BLUE)" => "<a href=\'ids.cgi\'>$Lang::tr{'intrusion detection system'} (BLUE)</a>" + "$Lang::tr{'intrusion detection system'} (BLUE)" => "<a href=\'ids.cgi\'>$Lang::tr{'intrusion detection system'} (BLUE)</a>", + "$Lang::tr{'random number generator daemon'}" => "<a href=\'entropy.cgi\'>$Lang::tr{'random number generator daemon'}</a>" ); my $lines=0; # Used to count the outputlines to make different bgcolor diff --git a/src/initscripts/system/rngd b/src/initscripts/system/rngd index df4aa7da2..9abcf6a79 100644 --- a/src/initscripts/system/rngd +++ b/src/initscripts/system/rngd @@ -30,10 +30,16 @@ case "${1}" in boot_mesg "Starting Random Number Generator Daemon..." if pidofproc /usr/sbin/rngd &>/dev/null; then + # Make sure PID file is readable. + test -f /run/rngd.pid && chmod 755 /run/rngd.pid + # Is already running. echo_ok else loadproc /usr/sbin/rngd --no-tpm=1 + + # Make sure PID file is readable. + test -f /run/rngd.pid && chmod 755 /run/rngd.pid fi ;; -- 2.13.6

2 1

[PATCH v2] WebUI: print more information on used nameservers
by Peter Müller 29 Jan '18

29 Jan '18

Show rDNS/PTR entry and country flag of the used nameservers on the nameserver status list at "netexternal.cgi". These information might be useful for debugging, or are just "nice to have" and do not harm anything. :-) Changes since v1: Corrected file link to country.cgi and added title to flag. Please disregard the first version of this patch. Signed-off-by: Peter Müller <peter.mueller(a)link38.eu> --- diff --git a/html/cgi-bin/netexternal.cgi b/html/cgi-bin/netexternal.cgi index 299612d4c..6fe0cc7d6 100644 --- a/html/cgi-bin/netexternal.cgi +++ b/html/cgi-bin/netexternal.cgi @@ -25,9 +25,13 @@ use strict; #use warnings; #use CGI::Carp 'fatalsToBrowser'; +use IO::Socket; +use Geo::IP::PurePerl; + require '/var/ipfire/general-functions.pl'; require "${General::swroot}/lang.pl"; require "${General::swroot}/header.pl"; +require "${General::swroot}/geoip-functions.pl"; require "${General::swroot}/graphs.pl"; my %color = (); @@ -99,6 +103,12 @@ if ( $querry[0] ne~ ""){ $Lang::tr{'nameserver'} </th> <th align="center"> + $Lang::tr{'flag'} + </th> + <th align="center"> + $Lang::tr{'rdns'} + </th> + <th align="center"> $Lang::tr{'status'} </th> </tr> @@ -139,9 +149,22 @@ END my $table_colour = ($id++ % 2) ? $color{'color22'} : $color{'color20'}; + # Get rDNS entry for nameserver (might be useful) + my $iaddr = inet_aton($nameserver); + my $rdns = gethostbyaddr($iaddr, AF_INET); + if (!$rdns) { $rdns = $Lang::tr{'lookup failed'}; } + + # Get country flag for the nameserver (might be useful) + my $gi = Geo::IP::PurePerl->new(); + my $ccode = $gi->country_code_by_name($nameserver); + my $fcode = lc($ccode); + my $flag_icon = &GeoIP::get_flag_icon($fcode); + print <<END; <tr bgcolor="$table_colour"> <td>$nameserver</td> + <td align="center"><a href="country.cgi#$fcode"><img src="$flag_icon" border="0" align="absmiddle" alt="$ccode" title="$ccode"></a></td> + <td>$rdns</td> <td bgcolor="$bgcolour" align="center"> $message </td> @@ -271,3 +294,4 @@ sub check_dnssec($$) { return $aware + $validating; } + diff --git a/langs/de/cgi-bin/de.pl b/langs/de/cgi-bin/de.pl index 0138cbc02..a63eb3160 100644 --- a/langs/de/cgi-bin/de.pl +++ b/langs/de/cgi-bin/de.pl @@ -1902,6 +1902,7 @@ 'quick playlist' => 'Quick Playlist', 'ram' => 'RAM-Speicher', 'random number generator daemon' => 'Random Number Generator Daemon', +'rdns' => 'rDNS', 'read bytes' => 'Gelesene Bytes', 'read list' => 'Liste der Leseberechtigten', 'real address' => 'Reale Addresse', diff --git a/langs/en/cgi-bin/en.pl b/langs/en/cgi-bin/en.pl index b3aee5a2b..13c18f4c8 100644 --- a/langs/en/cgi-bin/en.pl +++ b/langs/en/cgi-bin/en.pl @@ -1936,6 +1936,7 @@ 'quick playlist' => 'Quick Playlist', 'ram' => 'RAM', 'random number generator daemon' => 'Random Number Generator Daemon', +'rdns' => 'rDNS', 'read bytes' => 'Read Bytes', 'read list' => 'list with readonly hosts', 'real address' => 'Real Address',

4 16

Upgrading to OpenSSL 1.1.0
by Michael Tremer 18 Jan '18

18 Jan '18

Hello, I have started working on upgrading the entire distribution to OpenSSL 1.1.0. This is however not the easiest task since many packages are just incompatible with the API changes of OpenSSL. Therefore, I started this in an own branch, upgraded all sorts of packages that won't build and patched those who could be patched. However, this is still quite chaotic and I need some help of the maintainers of some of the packages to do this for their own packages. I have already dropped some packages in this process that a) were incompatible with OpenSSL 1.1.0, b) where no patches were available and c) that are not maintained upstream any longer. I also cherry-picked those commits to the current next tree. If someone disagrees, please open a separate discussion. The packages dropped are: * Pound * vsftp * sslscan Packages which currently don't build and I could not patch very easily: * php * asterisk * openvpn I suppose Erik is best to upgrade to openvpn 2.4, Dirk upgrades asterisk and I am quite sure that there is a few people out there who have been working on php. Please raise your hands. I would like to have the openssl 1.1 branch ready for merge into next at the end of December. Please make sure that any patches have been submitted until then. Please work on top of this branch: https://git.ipfire.org/pub/git/people/ms/ipfire-2.x.git openssl-11 https://git.ipfire.org/?p=people/ms/ipfire-2.x.git;a=shortlog;h=refs/heads/… Please also submit improvements of other packages that we can make sure of (i.e. better cipher suites for Apache, etc.)... Best, -Michael

6 20

[PATCH] Fix bug 10504: match download's sourceurl mangling in, updxlrator
by Justin Luth 07 Jan '18

07 Jan '18

Updatexlrator stores its files in a hash of the URL. The download utility mangles the URL for [+/~], but the updxlrator only does it for [/]. Thus, download stores the result as one hash, and updxlrator looks for it with a different hash. The result is that the file is re-downloaded every time by both the client, and updxlrator. This is fixed by making updxlrator mangle the url in the same way as the downloader. apt-get install g++ would be a good test for this. Signed-off-by: Justin Luth <jluth(a)mail.com> --- I submitted the bug report and attached this patch three years ago, but the maintainer of updxlrator - although he incorporated it into his own ipcop packagea - has never released it (for ipcop) or any of the other promised updates that he has been working on (in ipfire). I have a few more fixes for updxlrator that I want to submit, if this process goes well. --- config/updxlrator/updxlrator | 2 ++ 1 file changed, 2 insertions(+) diff --git a/config/updxlrator/updxlrator b/config/updxlrator/updxlrator index 2ddc6d8e4..b728902f6 100644 --- a/config/updxlrator/updxlrator +++ b/config/updxlrator/updxlrator @@ -345,7 +345,9 @@ sub check_cache my $sourceurl=$_[0]; my $cfmirror=$_[4]; + $sourceurl =~ s@\%2b@+@ig; $sourceurl =~ s@\%2f@/@ig; + $sourceurl =~ s@\%7e@~@ig; $updfile = substr($sourceurl,rindex($sourceurl,"/")+1); $updfile =~ s@\%20@ @ig; -- 2.14.1

2 3

[PATCH] Fix bug 11558 updxlrator: use mirror mode for SHA1, filenames
by Justin Luth 30 Dec '17

30 Dec '17

Most Microsoft updates now contain an SHA1 hash in the filename. Since these files are uniquely identifiable, use mirror mode (which creates a hash of just the filename instead of the entire URL) to cache them. (But first check the URL cache to see if it has been downloaded as a URL already.) This is a HUGELY needed fix. Windows 10 updates are 5+ GB per month, and we lose several days of bandwidth downloading duplicates from different mirrors. Sometimes a single client will request the same patch from multiple mirrors. That's bad. This patch will save a ton of bandwidth, and lots of disk space. The patch limits the SHA1 test to microsoft only, but it could be easily expanded to other vendors if there is a need. Signed-off-by: Justin Luth <jluth(a)mail.com> --- This is a slight hack, because the fix is tucked away in a somewhat obscure function. I mean, someone could completely redesign this and make more modular functions that create the hash, check if the file exists, etc. But this patch very neatly is contained in one section of the code and doesn't modify anything else, so I think the simplicity and elegance warrant the hackiness. Because the fix is tucked away in the check_cache function, I added one comment in the Microsoft section, clearly alerting future programmers about the change. Originally, I had put my SHA1 test here, but doing so required pre-processing the caches and renaming the hash identifiers. This patch avoids that ugly business. This patch works beautifully because it never downloads anything extra. If you already cached the URL, then you won't re-download the filename. But if you hit a different mirror now, you will download one more time (as normal) and after that every different mirror will be "satisfied". In the bug report, there is a script that can be tweaked to RENAME the URL hash to become a filename hash, in case any site really wants to avoid that possibility of redownloading a file they already have. But since I haven't seen anyone else complaining about this problem, I doubt anyone would be interested. A good test URL (that is a small file, not 1+ GB) is 7.au.download.windowsupdate.com/d/msdownload/update/others/2015/03/16743052… You can use numbers other than 7, remove the 7. or remove 7.au. in order to access different mirrors of the same file. --- config/updxlrator/updxlrator | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/config/updxlrator/updxlrator b/config/updxlrator/updxlrator index 5baaaae58..ff23b3a95 100644 --- a/config/updxlrator/updxlrator +++ b/config/updxlrator/updxlrator @@ -86,6 +86,8 @@ while (<>) { && ($source_url !~ m@\&@) ) { + # NOTE: check_cache will change to $mirror instead of $unique if the filename contains an SHA1 hash + # and the URL is not found in cache! $xlrator_url = &check_cache($source_url,$hostaddr,$username,"Microsoft",$unique); } @@ -400,6 +402,17 @@ sub check_cache &debuglog("Retrieving file from cache ($updsource) for $hostaddr"); &setcachestatus("$updcachedir/$vendorid/$uuid/access.log",time); $cacheurl="http://$netsettings{'GREEN_ADDRESS'}:$http_port/updatecache/$vendorid/$uuid/$updfile"; + } + elsif ( + ($cfmirror == $unique) && + ($vendorid == "microsoft") && + ($source_url =~ m@.*[0-9a-f]{40}\.[^\.]+@i) + ) + { + # Most Microsoft updates now have an SHA1 hash in the name. These should be treated as unique files. + # Since it wasn't found in the URL cache, switch to mirror mode and try again using just the filename. + &debuglog("SHA1: $vendorid $uuid not cached. Reprocessing as mirror $sourceurl"); + $cacheurl = &check_cache($source_url,$hostaddr,$username,$vendorid,$mirror); } else { -- 2.14.1

1 0

[PATCH] Fix bug 11567 updxlrator: don't prematurely release lock, file
by Justin Luth 30 Dec '17

30 Dec '17

With Microsoft's new style of downloading updates, where portions of a patch are requested multiple times per second, it has become extremely common for downloads to reach > 100%. Due to an early unlinking of the "lock" file, there is a big window of opportunity (between the unlink and wget actually saving some data) for multiple download/wget threads to start, adding to the same file. So not only is bandwidth wasted by duplicate downloads running simultaneously, but the resulting file is corrupt anyway. The problem is noticed more often by low bandwidth users (who need the benefits of updxlrator the most) because then wget's latency is even longer, creating a very wide window of opportunity. Ultimately, this needs something like "flock", where the file is set and tested in one operation. But for now, settle with the current test / create lock solution, and just stop unnecessarily releasing the lock. Since the file already exists as a lock when wget starts, wget now must ALWAYS run with --continue, which works fine on a zero-sized file. Signed-off-by: Justin Luth <jluth(a)mail.com> --- config/updxlrator/download | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/config/updxlrator/download b/config/updxlrator/download index dbc722c23..afa6e6cb9 100644 --- a/config/updxlrator/download +++ b/config/updxlrator/download @@ -30,7 +30,6 @@ my $unique=0; my $mirror=1; my %dlinfo=(); -my $wgetContinueFlag=""; my $vendorid = $ARGV[0]; if (!defined($vendorid) || $vendorid eq '') { exit; } my $sourceurl = $ARGV[1]; if (!defined($sourceurl) || $sourceurl eq '') { exit; } @@ -57,16 +56,15 @@ if($restartdl == 0) # this is a new download exit if (-e "$repository/download/$vendorid/$updatefile"); - # dotzball: Why is this necessary? + # hinder multiple downloads from starting simultaneously. Create empty "lock" file. + # TODO: Another thread may sneak in between these two commands - so not fool-proof, but good enough? system("touch $repository/download/$vendorid/$updatefile"); - $wgetContinueFlag = "-nc"; } else { # this is a restart of a previous (unfinished) download # -> continue download - $wgetContinueFlag = "-c"; &writelog("Continue download: $updatefile"); } @@ -133,7 +131,9 @@ unless($restartdl) { # this is a new download # -> download from scratch - unlink "$repository/download/$vendorid/$updatefile"; + + #already exited earlier if the file existed, and afterwards created this empty "lock", so if not empty now, another thread is already downloading it. + exit if ( -s "$repository/download/$vendorid/$updatefile" ); unlink "$repository/download/$vendorid/$updatefile.info"; } @@ -147,7 +147,7 @@ $dlinfo{'REMOTESIZE'} = $remote_size; $dlinfo{'STATUS'} = "1"; &UPDXLT::writehash("$repository/download/$vendorid/$updatefile.info", \%dlinfo); -my $cmd = "$UPDXLT::wget $login $dlrate --user-agent=\"$UPDXLT::useragent\" -q -P $repository/download/$vendorid $wgetContinueFlag $sourceurl"; +my $cmd = "$UPDXLT::wget $login $dlrate --user-agent=\"$UPDXLT::useragent\" -q -P $repository/download/$vendorid --continue $sourceurl"; $_ = system("$cmd"); $ENV{'http_proxy'} = ''; -- 2.14.1

2 1

[PATCH] updxlrator: show hostaddr in debuglog
by Justin Luth 30 Dec '17

30 Dec '17

There is nowhere in the debuglog any indication of which client is requesting the file that updxlrator is providing (or caching). Especially for those huge Windows 10 downloads, it is valuable to see which client is requesting them, especially when the same client requests the same download multiple times a second. This only impacts users who turn on debugging. Signed-off-by: Justin Luth <jluth(a)mail.com> --- config/updxlrator/updxlrator | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/config/updxlrator/updxlrator b/config/updxlrator/updxlrator index b728902f6..5baaaae58 100644 --- a/config/updxlrator/updxlrator +++ b/config/updxlrator/updxlrator @@ -397,7 +397,7 @@ sub check_cache ) { &debuglog("File exists in cache and is up to date"); - &debuglog("Retrieving file from cache ($updsource)"); + &debuglog("Retrieving file from cache ($updsource) for $hostaddr"); &setcachestatus("$updcachedir/$vendorid/$uuid/access.log",time); $cacheurl="http://$netsettings{'GREEN_ADDRESS'}:$http_port/updatecache/$vendorid/$uuid/$updfile"; } @@ -414,7 +414,7 @@ sub check_cache &debuglog("Free disk space: " . &diskfree($updcachedir)); &debuglog("Disk usage: " . &diskusage($updcachedir) . "% (max. $maxusage%)"); if (-e "$updcachedir/download/$vendorid/$updfile") { &debuglog("File download/$vendorid/$updfile exists"); } - &debuglog("Retrieving file from source ($updsource)"); + &debuglog("Retrieving file from source ($updsource) for $hostaddr"); if ((!$passive_mode) && (&diskusage($updcachedir) <= $maxusage) && ($remote_size <= &diskfree($updcachedir)) && (!-e "$updcachedir/download/$vendorid/$updfile")) { &debuglog("Running command $nice$apphome/bin/download $vendorid $sourceurl $cfmirror &"); -- 2.14.1

2 1

[PATCH] urlfilter.dat: Added Older/Newer links at top of page
by Matthias Fischer 26 Dec '17

26 Dec '17

Hi, Triggered by: https://forum.ipfire.org/viewtopic.php?f=4&t=19998#p112930 Added 'Older'/'Newer'-links for better page browsing. Best, Matthias Signed-off-by: Matthias Fischer <matthias.fischer(a)ipfire.org> --- html/cgi-bin/logs.cgi/urlfilter.dat | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/html/cgi-bin/logs.cgi/urlfilter.dat b/html/cgi-bin/logs.cgi/urlfilter.dat index ce642e1cb..221ed74ac 100644 --- a/html/cgi-bin/logs.cgi/urlfilter.dat +++ b/html/cgi-bin/logs.cgi/urlfilter.dat @@ -304,6 +304,10 @@ if ($cgiparams{'LOGTYPE'} eq 'urlfilter') if ($logsettings{'LOGVIEW_REVERSE'} eq 'on') { @slice = reverse @slice; } +if ($lines != 0) { + +&oldernewer(); + if ($lines) { @@ -369,7 +373,7 @@ if ($lines) print "</table> \n"; } - +} &oldernewer(); &Header::closebox(); -- 2.15.1

1 0

Merry Christmas & A Happy New Year!
by Michael Tremer 26 Dec '17

26 Dec '17

Hello fellow developers, 2017 has been a wonderful year for the project and me. I feel that it has been one of our most successful ones, yet. Although we do not have a very long list of features that we released, we got a few very good ones. But what I found really great in the last couple of months especially is the work attitude that everyone is bringing towards the project, the enthusiasm to start new things, and that we have a growing team of people. That all gives me huge confidence that 2018 will be an even better year and I would like to thank each and everyone of you for your contributions in the past - no matter if they have been small or larger, or regularly or sometimes not so often - you are a part of the team that makes this project special. A Merry Christmas to all of you (who are celebrating it) and a Happy New Year! Best, -Michael

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

Development December 2017