Re: [PATCH] BUG 11786 - squid: Remove setting for filter processes the number of Squid processes

[Michael Tremer <michael.tremer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 7383 bytes --]

On Tue, 2018-10-30 at 15:37 +0100, Matthias Fischer wrote:
> On 30.10.2018 15:03, Michael Tremer wrote:
> > Hey,
> 
> Hello,
> 
> > do you have any idea why the redirector is busy?
> 
> Sorry: no. Long time ago, I searched for an optimal value but didn't
> find any useful links. Right now, I'm testing 'squid 4.4" and its always
> the same. Too few redirectors => warning.
> 
> I only found something like "If you see this warning, you should
> increase the number of child processes and restart (or reconfigure)
> Squid. If the queue size becomes twice the number of redirectors, Squid
> aborts with a fatal message."

The last bit is not very good. Why does squid not queue the requests long
enough? That means even CPU cores * 2 or 4 won't be enough. On a single-core
machine that would only be 2 requests. LOL

Looking at the options that we have (
http://www.squid-cache.org/Doc/config/url_rewrite_children/), could you please
test this:

  url_rewrite_children X startup=N idle=N queue-size=64

With N being the number of cores that your system has. I do not think that there
is any point in trying to run more than N queries at a time.

But squid doesn't start anything when it starts up. startup= changes that and
idle= makes sure that all processes remain active.

A queue size of 64 seems sensible to me because that is probably what the
average tabloid newspaper page has. Maybe it should be 16 * N or so...

-Michael

> 
> Taken from:
> 
http://etutorials.org/Server+Administration/Squid.+The+definitive+guide/Chapter+11.+Redirectors/11.4+Configuring+Squid/
> 
> So my current "40"-value was found through "try and look if the warning
> comes again"...
> 
> Best,
> Matthias
> 
> 
> > -Michael
> > 
> > On Tue, 2018-10-30 at 14:54 +0100, Matthias Fischer wrote:
> > > Hi,
> > > 
> > > I tested this patch today and I fear the resulting value is too low.
> > > 
> > > For me, its leads to:
> > > 
> > > ***SNIP***
> > > ...
> > > 2018/10/30 14:42:43 kid1| WARNING: All 2/2 redirector processes are busy.
> > > 2018/10/30 14:42:43 kid1| WARNING: 4 pending requests queued
> > > 2018/10/30 14:42:43 kid1| WARNING: Consider increasing the number of
> > > redirector processes in your config file.
> > > ...
> > > 
> > > Current setting is "40" (Duo box, 2 Cores, 'squidclamav, squidguard)
> > > 
> > > Best,
> > > Matthias
> > > 
> > > On 30.10.2018 12:06, Daniel Weismüller wrote:
> > > > I added a function to determine the number of cores.
> > > > Now the number of squid processes will be equal to the number of logical
> > > > cores.
> > > > Further I removed the possibility of changing the number
> > > > of squid processes in the proxy.cgi
> > > > 
> > > > Signed-off-by: Daniel Weismüller <daniel.weismueller(a)ipfire.org>
> > > > ---
> > > >  config/cfgroot/general-functions.pl |  7 +++++++
> > > >  html/cgi-bin/proxy.cgi              | 16 ++--------------
> > > >  2 files changed, 9 insertions(+), 14 deletions(-)
> > > > 
> > > > diff --git a/config/cfgroot/general-functions.pl
> > > > b/config/cfgroot/general-
> > > > functions.pl
> > > > index 0577afe28..e8495e885 100644
> > > > --- a/config/cfgroot/general-functions.pl
> > > > +++ b/config/cfgroot/general-functions.pl
> > > > @@ -1165,5 +1165,12 @@ sub dnssec_status() {
> > > >  
> > > >  	return $status;
> > > >  }
> > > > +sub number_cpu_cores() {
> > > > +	open my $cpuinfo, "/proc/cpuinfo" or die "Can't open cpuinfo:
> > > > $!\n";
> > > > +	my $cores = scalar (map /^processor/, <$cpuinfo>);
> > > > +	close $cpuinfo;
> > > > +
> > > > +	return $cores;
> > > > +}
> > > >  
> > > >  1;
> > > > diff --git a/html/cgi-bin/proxy.cgi b/html/cgi-bin/proxy.cgi
> > > > index 738425b9a..92bebfe18 100644
> > > > --- a/html/cgi-bin/proxy.cgi
> > > > +++ b/html/cgi-bin/proxy.cgi
> > > > @@ -287,7 +287,6 @@ $proxysettings{'IDENT_USER_ACL'} = 'positive';
> > > >  $proxysettings{'ENABLE_FILTER'} = 'off';
> > > >  $proxysettings{'ENABLE_UPDXLRATOR'} = 'off';
> > > >  $proxysettings{'ENABLE_CLAMAV'} = 'off';
> > > > -$proxysettings{'CHILDREN'} = '10';
> > > >  
> > > >  $ncsa_buttontext = $Lang::tr{'advproxy NCSA create user'};
> > > >  
> > > > @@ -437,11 +436,6 @@ if (($proxysettings{'ACTION'} eq $Lang::tr{'save'})
> > > > ||
> > > > ($proxysettings{'ACTION'}
> > > >  		$errormessage = $Lang::tr{'invalid maximum incoming
> > > > size'};
> > > >  		goto ERROR;
> > > >  	}
> > > > -		if (!($proxysettings{'CHILDREN'} =~ /^\d+$/) ||
> > > > ($proxysettings{'CHILDREN'} < 1))
> > > > -	{
> > > > -		$errormessage = $Lang::tr{'advproxy invalid num of
> > > > children'};
> > > > -		goto ERROR;
> > > > -	}
> > > >  	if ($proxysettings{'ENABLE_BROWSER_CHECK'} eq 'on')
> > > >  	{
> > > >  		$browser_regexp = '';
> > > > @@ -1034,12 +1028,8 @@ print <<END
> > > >  </table>
> > > >  <hr size='1'>
> > > >  <table width='100%'>
> > > > -<tr><td class='base' colspan='4'><b>$Lang::tr{'advproxy redirector
> > > > children'}</b></td></tr>
> > > > -<tr><td class='base' >$Lang::tr{'processes'}:&nbsp;<img src='/blob.gif'
> > > > alt='*' /><input type='text' name='CHILDREN'
> > > > value='$proxysettings{'CHILDREN'}' size='5' /></td>
> > > >  END
> > > >  ;
> > > > -my $count = `ip n| wc -l`;
> > > > -if ( $count < 1 ){$count = 1;}
> > > >  if ( -e "/usr/bin/squidclamav" ) {
> > > >  	print "<td class='base'><b>".$Lang::tr{'advproxy
> > > > squidclamav'}."</b><br
> > > > />";
> > > >  	if ( ! -e "/var/run/clamav/clamd.pid" ){
> > > > @@ -1048,18 +1038,16 @@ if ( -e "/usr/bin/squidclamav" ) {
> > > >  		}
> > > >  	else {
> > > >  		print $Lang::tr{'advproxy enabled'}."<input
> > > > type='checkbox'
> > > > name='ENABLE_CLAMAV' ".$checked{'ENABLE_CLAMAV'}{'on'}." /><br />";
> > > > -		print "+ ".int(( $count**(1/3)) * 8);}
> > > > +}
> > > >  	print "</td>";
> > > >  } else {
> > > >  	print "<td></td>";
> > > >  }
> > > >  print "<td class='base'><a href='/cgi-
> > > > bin/urlfilter.cgi'><b>".$Lang::tr{'advproxy url filter'}."</a></b><br
> > > > />";
> > > >  print $Lang::tr{'advproxy enabled'}."<input type='checkbox'
> > > > name='ENABLE_FILTER' ".$checked{'ENABLE_FILTER'}{'on'}." /><br />";
> > > > -print "+ ".int(($count**(1/3)) * 6);
> > > >  print "</td>";
> > > >  print "<td class='base'><a href='/cgi-
> > > > bin/updatexlrator.cgi'><b>".$Lang::tr{'advproxy update
> > > > accelerator'}."</a></b><br />";
> > > >  print $Lang::tr{'advproxy enabled'}."<input type='checkbox'
> > > > name='ENABLE_UPDXLRATOR' ".$checked{'ENABLE_UPDXLRATOR'}{'on'}." /><br
> > > > />";
> > > > -print "+ ".int(($count**(1/3)) * 5);
> > > >  print "</td></tr>";
> > > >  print <<END
> > > >  </table>
> > > > @@ -4095,7 +4083,7 @@ END
> > > >  	if (($proxysettings{'ENABLE_FILTER'} eq 'on') ||
> > > > ($proxysettings{'ENABLE_UPDXLRATOR'} eq 'on') ||
> > > > ($proxysettings{'ENABLE_CLAMAV'} eq 'on'))
> > > >  	{
> > > >  		print FILE "url_rewrite_program
> > > > /usr/sbin/redirect_wrapper\n";
> > > > -		print FILE "url_rewrite_children
> > > > $proxysettings{'CHILDREN'}\n\n";
> > > > +		print FILE "url_rewrite_children ",
> > > > &General::number_cpu_cores(), "\n\n";
> > > >  	}
> > > >  
> > > >  	# Include file with user defined settings.
> > > > 
> > > 
> > > 
> > 
> > 
> 
>