[PATCH] BUG 11786 - squid: Remove setting for filter processes the number of Squid processes

[PATCH] BUG 11786 - squid: Remove setting for filter processes the number of Squid processes

[Daniel Weismüller]

[-- Attachment #1: Type: text/plain, Size: 3878 bytes --]

I added a function to determine the number of cores.
Now the number of squid processes will be equal to the number of logical cores.
Further I removed the possibility of changing the number
of squid processes in the proxy.cgi

Signed-off-by: Daniel Weismüller <daniel.weismueller(a)ipfire.org>
---
 config/cfgroot/general-functions.pl |  7 +++++++
 html/cgi-bin/proxy.cgi              | 16 ++--------------
 2 files changed, 9 insertions(+), 14 deletions(-)

diff --git a/config/cfgroot/general-functions.pl b/config/cfgroot/general-functions.pl
index 0577afe28..e8495e885 100644
--- a/config/cfgroot/general-functions.pl
+++ b/config/cfgroot/general-functions.pl
@@ -1165,5 +1165,12 @@ sub dnssec_status() {
 
 	return $status;
 }
+sub number_cpu_cores() {
+	open my $cpuinfo, "/proc/cpuinfo" or die "Can't open cpuinfo: $!\n";
+	my $cores = scalar (map /^processor/, <$cpuinfo>);
+	close $cpuinfo;
+
+	return $cores;
+}
 
 1;
diff --git a/html/cgi-bin/proxy.cgi b/html/cgi-bin/proxy.cgi
index 738425b9a..92bebfe18 100644
--- a/html/cgi-bin/proxy.cgi
+++ b/html/cgi-bin/proxy.cgi
@@ -287,7 +287,6 @@ $proxysettings{'IDENT_USER_ACL'} = 'positive';
 $proxysettings{'ENABLE_FILTER'} = 'off';
 $proxysettings{'ENABLE_UPDXLRATOR'} = 'off';
 $proxysettings{'ENABLE_CLAMAV'} = 'off';
-$proxysettings{'CHILDREN'} = '10';
 
 $ncsa_buttontext = $Lang::tr{'advproxy NCSA create user'};
 
@@ -437,11 +436,6 @@ if (($proxysettings{'ACTION'} eq $Lang::tr{'save'}) || ($proxysettings{'ACTION'}
 		$errormessage = $Lang::tr{'invalid maximum incoming size'};
 		goto ERROR;
 	}
-		if (!($proxysettings{'CHILDREN'} =~ /^\d+$/) || ($proxysettings{'CHILDREN'} < 1))
-	{
-		$errormessage = $Lang::tr{'advproxy invalid num of children'};
-		goto ERROR;
-	}
 	if ($proxysettings{'ENABLE_BROWSER_CHECK'} eq 'on')
 	{
 		$browser_regexp = '';
@@ -1034,12 +1028,8 @@ print <<END
 </table>
 <hr size='1'>
 <table width='100%'>
-<tr><td class='base' colspan='4'><b>$Lang::tr{'advproxy redirector children'}</b></td></tr>
-<tr><td class='base' >$Lang::tr{'processes'}:&nbsp;<img src='/blob.gif' alt='*' /><input type='text' name='CHILDREN' value='$proxysettings{'CHILDREN'}' size='5' /></td>
 END
 ;
-my $count = `ip n| wc -l`;
-if ( $count < 1 ){$count = 1;}
 if ( -e "/usr/bin/squidclamav" ) {
 	print "<td class='base'><b>".$Lang::tr{'advproxy squidclamav'}."</b><br />";
 	if ( ! -e "/var/run/clamav/clamd.pid" ){
@@ -1048,18 +1038,16 @@ if ( -e "/usr/bin/squidclamav" ) {
 		}
 	else {
 		print $Lang::tr{'advproxy enabled'}."<input type='checkbox' name='ENABLE_CLAMAV' ".$checked{'ENABLE_CLAMAV'}{'on'}." /><br />";
-		print "+ ".int(( $count**(1/3)) * 8);}
+}
 	print "</td>";
 } else {
 	print "<td></td>";
 }
 print "<td class='base'><a href='/cgi-bin/urlfilter.cgi'><b>".$Lang::tr{'advproxy url filter'}."</a></b><br />";
 print $Lang::tr{'advproxy enabled'}."<input type='checkbox' name='ENABLE_FILTER' ".$checked{'ENABLE_FILTER'}{'on'}." /><br />";
-print "+ ".int(($count**(1/3)) * 6);
 print "</td>";
 print "<td class='base'><a href='/cgi-bin/updatexlrator.cgi'><b>".$Lang::tr{'advproxy update accelerator'}."</a></b><br />";
 print $Lang::tr{'advproxy enabled'}."<input type='checkbox' name='ENABLE_UPDXLRATOR' ".$checked{'ENABLE_UPDXLRATOR'}{'on'}." /><br />";
-print "+ ".int(($count**(1/3)) * 5);
 print "</td></tr>";
 print <<END
 </table>
@@ -4095,7 +4083,7 @@ END
 	if (($proxysettings{'ENABLE_FILTER'} eq 'on') || ($proxysettings{'ENABLE_UPDXLRATOR'} eq 'on') || ($proxysettings{'ENABLE_CLAMAV'} eq 'on'))
 	{
 		print FILE "url_rewrite_program /usr/sbin/redirect_wrapper\n";
-		print FILE "url_rewrite_children $proxysettings{'CHILDREN'}\n\n";
+		print FILE "url_rewrite_children ", &General::number_cpu_cores(), "\n\n";
 	}
 
 	# Include file with user defined settings.
-- 
2.12.2

Re: [PATCH] BUG 11786 - squid: Remove setting for filter processes the number of Squid processes

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 4574 bytes --]

Hi,

I tested this patch today and I fear the resulting value is too low.

For me, its leads to:

***SNIP***
...
2018/10/30 14:42:43 kid1| WARNING: All 2/2 redirector processes are busy.
2018/10/30 14:42:43 kid1| WARNING: 4 pending requests queued
2018/10/30 14:42:43 kid1| WARNING: Consider increasing the number of
redirector processes in your config file.
...

Current setting is "40" (Duo box, 2 Cores, 'squidclamav, squidguard)

Best,
Matthias

On 30.10.2018 12:06, Daniel Weismüller wrote:
> I added a function to determine the number of cores.
> Now the number of squid processes will be equal to the number of logical cores.
> Further I removed the possibility of changing the number
> of squid processes in the proxy.cgi
> 
> Signed-off-by: Daniel Weismüller <daniel.weismueller(a)ipfire.org>
> ---
>  config/cfgroot/general-functions.pl |  7 +++++++
>  html/cgi-bin/proxy.cgi              | 16 ++--------------
>  2 files changed, 9 insertions(+), 14 deletions(-)
> 
> diff --git a/config/cfgroot/general-functions.pl b/config/cfgroot/general-functions.pl
> index 0577afe28..e8495e885 100644
> --- a/config/cfgroot/general-functions.pl
> +++ b/config/cfgroot/general-functions.pl
> @@ -1165,5 +1165,12 @@ sub dnssec_status() {
>  
>  	return $status;
>  }
> +sub number_cpu_cores() {
> +	open my $cpuinfo, "/proc/cpuinfo" or die "Can't open cpuinfo: $!\n";
> +	my $cores = scalar (map /^processor/, <$cpuinfo>);
> +	close $cpuinfo;
> +
> +	return $cores;
> +}
>  
>  1;
> diff --git a/html/cgi-bin/proxy.cgi b/html/cgi-bin/proxy.cgi
> index 738425b9a..92bebfe18 100644
> --- a/html/cgi-bin/proxy.cgi
> +++ b/html/cgi-bin/proxy.cgi
> @@ -287,7 +287,6 @@ $proxysettings{'IDENT_USER_ACL'} = 'positive';
>  $proxysettings{'ENABLE_FILTER'} = 'off';
>  $proxysettings{'ENABLE_UPDXLRATOR'} = 'off';
>  $proxysettings{'ENABLE_CLAMAV'} = 'off';
> -$proxysettings{'CHILDREN'} = '10';
>  
>  $ncsa_buttontext = $Lang::tr{'advproxy NCSA create user'};
>  
> @@ -437,11 +436,6 @@ if (($proxysettings{'ACTION'} eq $Lang::tr{'save'}) || ($proxysettings{'ACTION'}
>  		$errormessage = $Lang::tr{'invalid maximum incoming size'};
>  		goto ERROR;
>  	}
> -		if (!($proxysettings{'CHILDREN'} =~ /^\d+$/) || ($proxysettings{'CHILDREN'} < 1))
> -	{
> -		$errormessage = $Lang::tr{'advproxy invalid num of children'};
> -		goto ERROR;
> -	}
>  	if ($proxysettings{'ENABLE_BROWSER_CHECK'} eq 'on')
>  	{
>  		$browser_regexp = '';
> @@ -1034,12 +1028,8 @@ print <<END
>  </table>
>  <hr size='1'>
>  <table width='100%'>
> -<tr><td class='base' colspan='4'><b>$Lang::tr{'advproxy redirector children'}</b></td></tr>
> -<tr><td class='base' >$Lang::tr{'processes'}:&nbsp;<img src='/blob.gif' alt='*' /><input type='text' name='CHILDREN' value='$proxysettings{'CHILDREN'}' size='5' /></td>
>  END
>  ;
> -my $count = `ip n| wc -l`;
> -if ( $count < 1 ){$count = 1;}
>  if ( -e "/usr/bin/squidclamav" ) {
>  	print "<td class='base'><b>".$Lang::tr{'advproxy squidclamav'}."</b><br />";
>  	if ( ! -e "/var/run/clamav/clamd.pid" ){
> @@ -1048,18 +1038,16 @@ if ( -e "/usr/bin/squidclamav" ) {
>  		}
>  	else {
>  		print $Lang::tr{'advproxy enabled'}."<input type='checkbox' name='ENABLE_CLAMAV' ".$checked{'ENABLE_CLAMAV'}{'on'}." /><br />";
> -		print "+ ".int(( $count**(1/3)) * 8);}
> +}
>  	print "</td>";
>  } else {
>  	print "<td></td>";
>  }
>  print "<td class='base'><a href='/cgi-bin/urlfilter.cgi'><b>".$Lang::tr{'advproxy url filter'}."</a></b><br />";
>  print $Lang::tr{'advproxy enabled'}."<input type='checkbox' name='ENABLE_FILTER' ".$checked{'ENABLE_FILTER'}{'on'}." /><br />";
> -print "+ ".int(($count**(1/3)) * 6);
>  print "</td>";
>  print "<td class='base'><a href='/cgi-bin/updatexlrator.cgi'><b>".$Lang::tr{'advproxy update accelerator'}."</a></b><br />";
>  print $Lang::tr{'advproxy enabled'}."<input type='checkbox' name='ENABLE_UPDXLRATOR' ".$checked{'ENABLE_UPDXLRATOR'}{'on'}." /><br />";
> -print "+ ".int(($count**(1/3)) * 5);
>  print "</td></tr>";
>  print <<END
>  </table>
> @@ -4095,7 +4083,7 @@ END
>  	if (($proxysettings{'ENABLE_FILTER'} eq 'on') || ($proxysettings{'ENABLE_UPDXLRATOR'} eq 'on') || ($proxysettings{'ENABLE_CLAMAV'} eq 'on'))
>  	{
>  		print FILE "url_rewrite_program /usr/sbin/redirect_wrapper\n";
> -		print FILE "url_rewrite_children $proxysettings{'CHILDREN'}\n\n";
> +		print FILE "url_rewrite_children ", &General::number_cpu_cores(), "\n\n";
>  	}
>  
>  	# Include file with user defined settings.
> 

Re: [PATCH] BUG 11786 - squid: Remove setting for filter processes the number of Squid processes

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 5037 bytes --]

Hey,

do you have any idea why the redirector is busy?

-Michael

On Tue, 2018-10-30 at 14:54 +0100, Matthias Fischer wrote:
> Hi,
> 
> I tested this patch today and I fear the resulting value is too low.
> 
> For me, its leads to:
> 
> ***SNIP***
> ...
> 2018/10/30 14:42:43 kid1| WARNING: All 2/2 redirector processes are busy.
> 2018/10/30 14:42:43 kid1| WARNING: 4 pending requests queued
> 2018/10/30 14:42:43 kid1| WARNING: Consider increasing the number of
> redirector processes in your config file.
> ...
> 
> Current setting is "40" (Duo box, 2 Cores, 'squidclamav, squidguard)
> 
> Best,
> Matthias
> 
> On 30.10.2018 12:06, Daniel Weismüller wrote:
> > I added a function to determine the number of cores.
> > Now the number of squid processes will be equal to the number of logical
> > cores.
> > Further I removed the possibility of changing the number
> > of squid processes in the proxy.cgi
> > 
> > Signed-off-by: Daniel Weismüller <daniel.weismueller(a)ipfire.org>
> > ---
> >  config/cfgroot/general-functions.pl |  7 +++++++
> >  html/cgi-bin/proxy.cgi              | 16 ++--------------
> >  2 files changed, 9 insertions(+), 14 deletions(-)
> > 
> > diff --git a/config/cfgroot/general-functions.pl b/config/cfgroot/general-
> > functions.pl
> > index 0577afe28..e8495e885 100644
> > --- a/config/cfgroot/general-functions.pl
> > +++ b/config/cfgroot/general-functions.pl
> > @@ -1165,5 +1165,12 @@ sub dnssec_status() {
> >  
> >  	return $status;
> >  }
> > +sub number_cpu_cores() {
> > +	open my $cpuinfo, "/proc/cpuinfo" or die "Can't open cpuinfo: $!\n";
> > +	my $cores = scalar (map /^processor/, <$cpuinfo>);
> > +	close $cpuinfo;
> > +
> > +	return $cores;
> > +}
> >  
> >  1;
> > diff --git a/html/cgi-bin/proxy.cgi b/html/cgi-bin/proxy.cgi
> > index 738425b9a..92bebfe18 100644
> > --- a/html/cgi-bin/proxy.cgi
> > +++ b/html/cgi-bin/proxy.cgi
> > @@ -287,7 +287,6 @@ $proxysettings{'IDENT_USER_ACL'} = 'positive';
> >  $proxysettings{'ENABLE_FILTER'} = 'off';
> >  $proxysettings{'ENABLE_UPDXLRATOR'} = 'off';
> >  $proxysettings{'ENABLE_CLAMAV'} = 'off';
> > -$proxysettings{'CHILDREN'} = '10';
> >  
> >  $ncsa_buttontext = $Lang::tr{'advproxy NCSA create user'};
> >  
> > @@ -437,11 +436,6 @@ if (($proxysettings{'ACTION'} eq $Lang::tr{'save'}) ||
> > ($proxysettings{'ACTION'}
> >  		$errormessage = $Lang::tr{'invalid maximum incoming size'};
> >  		goto ERROR;
> >  	}
> > -		if (!($proxysettings{'CHILDREN'} =~ /^\d+$/) ||
> > ($proxysettings{'CHILDREN'} < 1))
> > -	{
> > -		$errormessage = $Lang::tr{'advproxy invalid num of children'};
> > -		goto ERROR;
> > -	}
> >  	if ($proxysettings{'ENABLE_BROWSER_CHECK'} eq 'on')
> >  	{
> >  		$browser_regexp = '';
> > @@ -1034,12 +1028,8 @@ print <<END
> >  </table>
> >  <hr size='1'>
> >  <table width='100%'>
> > -<tr><td class='base' colspan='4'><b>$Lang::tr{'advproxy redirector
> > children'}</b></td></tr>
> > -<tr><td class='base' >$Lang::tr{'processes'}:&nbsp;<img src='/blob.gif'
> > alt='*' /><input type='text' name='CHILDREN'
> > value='$proxysettings{'CHILDREN'}' size='5' /></td>
> >  END
> >  ;
> > -my $count = `ip n| wc -l`;
> > -if ( $count < 1 ){$count = 1;}
> >  if ( -e "/usr/bin/squidclamav" ) {
> >  	print "<td class='base'><b>".$Lang::tr{'advproxy squidclamav'}."</b><br
> > />";
> >  	if ( ! -e "/var/run/clamav/clamd.pid" ){
> > @@ -1048,18 +1038,16 @@ if ( -e "/usr/bin/squidclamav" ) {
> >  		}
> >  	else {
> >  		print $Lang::tr{'advproxy enabled'}."<input type='checkbox'
> > name='ENABLE_CLAMAV' ".$checked{'ENABLE_CLAMAV'}{'on'}." /><br />";
> > -		print "+ ".int(( $count**(1/3)) * 8);}
> > +}
> >  	print "</td>";
> >  } else {
> >  	print "<td></td>";
> >  }
> >  print "<td class='base'><a href='/cgi-
> > bin/urlfilter.cgi'><b>".$Lang::tr{'advproxy url filter'}."</a></b><br />";
> >  print $Lang::tr{'advproxy enabled'}."<input type='checkbox'
> > name='ENABLE_FILTER' ".$checked{'ENABLE_FILTER'}{'on'}." /><br />";
> > -print "+ ".int(($count**(1/3)) * 6);
> >  print "</td>";
> >  print "<td class='base'><a href='/cgi-
> > bin/updatexlrator.cgi'><b>".$Lang::tr{'advproxy update
> > accelerator'}."</a></b><br />";
> >  print $Lang::tr{'advproxy enabled'}."<input type='checkbox'
> > name='ENABLE_UPDXLRATOR' ".$checked{'ENABLE_UPDXLRATOR'}{'on'}." /><br />";
> > -print "+ ".int(($count**(1/3)) * 5);
> >  print "</td></tr>";
> >  print <<END
> >  </table>
> > @@ -4095,7 +4083,7 @@ END
> >  	if (($proxysettings{'ENABLE_FILTER'} eq 'on') ||
> > ($proxysettings{'ENABLE_UPDXLRATOR'} eq 'on') ||
> > ($proxysettings{'ENABLE_CLAMAV'} eq 'on'))
> >  	{
> >  		print FILE "url_rewrite_program /usr/sbin/redirect_wrapper\n";
> > -		print FILE "url_rewrite_children
> > $proxysettings{'CHILDREN'}\n\n";
> > +		print FILE "url_rewrite_children ",
> > &General::number_cpu_cores(), "\n\n";
> >  	}
> >  
> >  	# Include file with user defined settings.
> > 
> 
> 

Re: [PATCH] BUG 11786 - squid: Remove setting for filter processes the number of Squid processes

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 5923 bytes --]

On 30.10.2018 15:03, Michael Tremer wrote:
> Hey,

Hello,

> do you have any idea why the redirector is busy?

Sorry: no. Long time ago, I searched for an optimal value but didn't
find any useful links. Right now, I'm testing 'squid 4.4" and its always
the same. Too few redirectors => warning.

I only found something like "If you see this warning, you should
increase the number of child processes and restart (or reconfigure)
Squid. If the queue size becomes twice the number of redirectors, Squid
aborts with a fatal message."

Taken from:
http://etutorials.org/Server+Administration/Squid.+The+definitive+guide/Chapter+11.+Redirectors/11.4+Configuring+Squid/

So my current "40"-value was found through "try and look if the warning
comes again"...

Best,
Matthias


> -Michael
> 
> On Tue, 2018-10-30 at 14:54 +0100, Matthias Fischer wrote:
>> Hi,
>> 
>> I tested this patch today and I fear the resulting value is too low.
>> 
>> For me, its leads to:
>> 
>> ***SNIP***
>> ...
>> 2018/10/30 14:42:43 kid1| WARNING: All 2/2 redirector processes are busy.
>> 2018/10/30 14:42:43 kid1| WARNING: 4 pending requests queued
>> 2018/10/30 14:42:43 kid1| WARNING: Consider increasing the number of
>> redirector processes in your config file.
>> ...
>> 
>> Current setting is "40" (Duo box, 2 Cores, 'squidclamav, squidguard)
>> 
>> Best,
>> Matthias
>> 
>> On 30.10.2018 12:06, Daniel Weismüller wrote:
>> > I added a function to determine the number of cores.
>> > Now the number of squid processes will be equal to the number of logical
>> > cores.
>> > Further I removed the possibility of changing the number
>> > of squid processes in the proxy.cgi
>> > 
>> > Signed-off-by: Daniel Weismüller <daniel.weismueller(a)ipfire.org>
>> > ---
>> >  config/cfgroot/general-functions.pl |  7 +++++++
>> >  html/cgi-bin/proxy.cgi              | 16 ++--------------
>> >  2 files changed, 9 insertions(+), 14 deletions(-)
>> > 
>> > diff --git a/config/cfgroot/general-functions.pl b/config/cfgroot/general-
>> > functions.pl
>> > index 0577afe28..e8495e885 100644
>> > --- a/config/cfgroot/general-functions.pl
>> > +++ b/config/cfgroot/general-functions.pl
>> > @@ -1165,5 +1165,12 @@ sub dnssec_status() {
>> >  
>> >  	return $status;
>> >  }
>> > +sub number_cpu_cores() {
>> > +	open my $cpuinfo, "/proc/cpuinfo" or die "Can't open cpuinfo: $!\n";
>> > +	my $cores = scalar (map /^processor/, <$cpuinfo>);
>> > +	close $cpuinfo;
>> > +
>> > +	return $cores;
>> > +}
>> >  
>> >  1;
>> > diff --git a/html/cgi-bin/proxy.cgi b/html/cgi-bin/proxy.cgi
>> > index 738425b9a..92bebfe18 100644
>> > --- a/html/cgi-bin/proxy.cgi
>> > +++ b/html/cgi-bin/proxy.cgi
>> > @@ -287,7 +287,6 @@ $proxysettings{'IDENT_USER_ACL'} = 'positive';
>> >  $proxysettings{'ENABLE_FILTER'} = 'off';
>> >  $proxysettings{'ENABLE_UPDXLRATOR'} = 'off';
>> >  $proxysettings{'ENABLE_CLAMAV'} = 'off';
>> > -$proxysettings{'CHILDREN'} = '10';
>> >  
>> >  $ncsa_buttontext = $Lang::tr{'advproxy NCSA create user'};
>> >  
>> > @@ -437,11 +436,6 @@ if (($proxysettings{'ACTION'} eq $Lang::tr{'save'}) ||
>> > ($proxysettings{'ACTION'}
>> >  		$errormessage = $Lang::tr{'invalid maximum incoming size'};
>> >  		goto ERROR;
>> >  	}
>> > -		if (!($proxysettings{'CHILDREN'} =~ /^\d+$/) ||
>> > ($proxysettings{'CHILDREN'} < 1))
>> > -	{
>> > -		$errormessage = $Lang::tr{'advproxy invalid num of children'};
>> > -		goto ERROR;
>> > -	}
>> >  	if ($proxysettings{'ENABLE_BROWSER_CHECK'} eq 'on')
>> >  	{
>> >  		$browser_regexp = '';
>> > @@ -1034,12 +1028,8 @@ print <<END
>> >  </table>
>> >  <hr size='1'>
>> >  <table width='100%'>
>> > -<tr><td class='base' colspan='4'><b>$Lang::tr{'advproxy redirector
>> > children'}</b></td></tr>
>> > -<tr><td class='base' >$Lang::tr{'processes'}:&nbsp;<img src='/blob.gif'
>> > alt='*' /><input type='text' name='CHILDREN'
>> > value='$proxysettings{'CHILDREN'}' size='5' /></td>
>> >  END
>> >  ;
>> > -my $count = `ip n| wc -l`;
>> > -if ( $count < 1 ){$count = 1;}
>> >  if ( -e "/usr/bin/squidclamav" ) {
>> >  	print "<td class='base'><b>".$Lang::tr{'advproxy squidclamav'}."</b><br
>> > />";
>> >  	if ( ! -e "/var/run/clamav/clamd.pid" ){
>> > @@ -1048,18 +1038,16 @@ if ( -e "/usr/bin/squidclamav" ) {
>> >  		}
>> >  	else {
>> >  		print $Lang::tr{'advproxy enabled'}."<input type='checkbox'
>> > name='ENABLE_CLAMAV' ".$checked{'ENABLE_CLAMAV'}{'on'}." /><br />";
>> > -		print "+ ".int(( $count**(1/3)) * 8);}
>> > +}
>> >  	print "</td>";
>> >  } else {
>> >  	print "<td></td>";
>> >  }
>> >  print "<td class='base'><a href='/cgi-
>> > bin/urlfilter.cgi'><b>".$Lang::tr{'advproxy url filter'}."</a></b><br />";
>> >  print $Lang::tr{'advproxy enabled'}."<input type='checkbox'
>> > name='ENABLE_FILTER' ".$checked{'ENABLE_FILTER'}{'on'}." /><br />";
>> > -print "+ ".int(($count**(1/3)) * 6);
>> >  print "</td>";
>> >  print "<td class='base'><a href='/cgi-
>> > bin/updatexlrator.cgi'><b>".$Lang::tr{'advproxy update
>> > accelerator'}."</a></b><br />";
>> >  print $Lang::tr{'advproxy enabled'}."<input type='checkbox'
>> > name='ENABLE_UPDXLRATOR' ".$checked{'ENABLE_UPDXLRATOR'}{'on'}." /><br />";
>> > -print "+ ".int(($count**(1/3)) * 5);
>> >  print "</td></tr>";
>> >  print <<END
>> >  </table>
>> > @@ -4095,7 +4083,7 @@ END
>> >  	if (($proxysettings{'ENABLE_FILTER'} eq 'on') ||
>> > ($proxysettings{'ENABLE_UPDXLRATOR'} eq 'on') ||
>> > ($proxysettings{'ENABLE_CLAMAV'} eq 'on'))
>> >  	{
>> >  		print FILE "url_rewrite_program /usr/sbin/redirect_wrapper\n";
>> > -		print FILE "url_rewrite_children
>> > $proxysettings{'CHILDREN'}\n\n";
>> > +		print FILE "url_rewrite_children ",
>> > &General::number_cpu_cores(), "\n\n";
>> >  	}
>> >  
>> >  	# Include file with user defined settings.
>> > 
>> 
>> 
> 
> 

Re: [PATCH] BUG 11786 - squid: Remove setting for filter processes the number of Squid processes

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 7383 bytes --]

On Tue, 2018-10-30 at 15:37 +0100, Matthias Fischer wrote:
> On 30.10.2018 15:03, Michael Tremer wrote:
> > Hey,
> 
> Hello,
> 
> > do you have any idea why the redirector is busy?
> 
> Sorry: no. Long time ago, I searched for an optimal value but didn't
> find any useful links. Right now, I'm testing 'squid 4.4" and its always
> the same. Too few redirectors => warning.
> 
> I only found something like "If you see this warning, you should
> increase the number of child processes and restart (or reconfigure)
> Squid. If the queue size becomes twice the number of redirectors, Squid
> aborts with a fatal message."

The last bit is not very good. Why does squid not queue the requests long
enough? That means even CPU cores * 2 or 4 won't be enough. On a single-core
machine that would only be 2 requests. LOL

Looking at the options that we have (
http://www.squid-cache.org/Doc/config/url_rewrite_children/), could you please
test this:

  url_rewrite_children X startup=N idle=N queue-size=64

With N being the number of cores that your system has. I do not think that there
is any point in trying to run more than N queries at a time.

But squid doesn't start anything when it starts up. startup= changes that and
idle= makes sure that all processes remain active.

A queue size of 64 seems sensible to me because that is probably what the
average tabloid newspaper page has. Maybe it should be 16 * N or so...

-Michael

> 
> Taken from:
> 
http://etutorials.org/Server+Administration/Squid.+The+definitive+guide/Chapter+11.+Redirectors/11.4+Configuring+Squid/
> 
> So my current "40"-value was found through "try and look if the warning
> comes again"...
> 
> Best,
> Matthias
> 
> 
> > -Michael
> > 
> > On Tue, 2018-10-30 at 14:54 +0100, Matthias Fischer wrote:
> > > Hi,
> > > 
> > > I tested this patch today and I fear the resulting value is too low.
> > > 
> > > For me, its leads to:
> > > 
> > > ***SNIP***
> > > ...
> > > 2018/10/30 14:42:43 kid1| WARNING: All 2/2 redirector processes are busy.
> > > 2018/10/30 14:42:43 kid1| WARNING: 4 pending requests queued
> > > 2018/10/30 14:42:43 kid1| WARNING: Consider increasing the number of
> > > redirector processes in your config file.
> > > ...
> > > 
> > > Current setting is "40" (Duo box, 2 Cores, 'squidclamav, squidguard)
> > > 
> > > Best,
> > > Matthias
> > > 
> > > On 30.10.2018 12:06, Daniel Weismüller wrote:
> > > > I added a function to determine the number of cores.
> > > > Now the number of squid processes will be equal to the number of logical
> > > > cores.
> > > > Further I removed the possibility of changing the number
> > > > of squid processes in the proxy.cgi
> > > > 
> > > > Signed-off-by: Daniel Weismüller <daniel.weismueller(a)ipfire.org>
> > > > ---
> > > >  config/cfgroot/general-functions.pl |  7 +++++++
> > > >  html/cgi-bin/proxy.cgi              | 16 ++--------------
> > > >  2 files changed, 9 insertions(+), 14 deletions(-)
> > > > 
> > > > diff --git a/config/cfgroot/general-functions.pl
> > > > b/config/cfgroot/general-
> > > > functions.pl
> > > > index 0577afe28..e8495e885 100644
> > > > --- a/config/cfgroot/general-functions.pl
> > > > +++ b/config/cfgroot/general-functions.pl
> > > > @@ -1165,5 +1165,12 @@ sub dnssec_status() {
> > > >  
> > > >  	return $status;
> > > >  }
> > > > +sub number_cpu_cores() {
> > > > +	open my $cpuinfo, "/proc/cpuinfo" or die "Can't open cpuinfo:
> > > > $!\n";
> > > > +	my $cores = scalar (map /^processor/, <$cpuinfo>);
> > > > +	close $cpuinfo;
> > > > +
> > > > +	return $cores;
> > > > +}
> > > >  
> > > >  1;
> > > > diff --git a/html/cgi-bin/proxy.cgi b/html/cgi-bin/proxy.cgi
> > > > index 738425b9a..92bebfe18 100644
> > > > --- a/html/cgi-bin/proxy.cgi
> > > > +++ b/html/cgi-bin/proxy.cgi
> > > > @@ -287,7 +287,6 @@ $proxysettings{'IDENT_USER_ACL'} = 'positive';
> > > >  $proxysettings{'ENABLE_FILTER'} = 'off';
> > > >  $proxysettings{'ENABLE_UPDXLRATOR'} = 'off';
> > > >  $proxysettings{'ENABLE_CLAMAV'} = 'off';
> > > > -$proxysettings{'CHILDREN'} = '10';
> > > >  
> > > >  $ncsa_buttontext = $Lang::tr{'advproxy NCSA create user'};
> > > >  
> > > > @@ -437,11 +436,6 @@ if (($proxysettings{'ACTION'} eq $Lang::tr{'save'})
> > > > ||
> > > > ($proxysettings{'ACTION'}
> > > >  		$errormessage = $Lang::tr{'invalid maximum incoming
> > > > size'};
> > > >  		goto ERROR;
> > > >  	}
> > > > -		if (!($proxysettings{'CHILDREN'} =~ /^\d+$/) ||
> > > > ($proxysettings{'CHILDREN'} < 1))
> > > > -	{
> > > > -		$errormessage = $Lang::tr{'advproxy invalid num of
> > > > children'};
> > > > -		goto ERROR;
> > > > -	}
> > > >  	if ($proxysettings{'ENABLE_BROWSER_CHECK'} eq 'on')
> > > >  	{
> > > >  		$browser_regexp = '';
> > > > @@ -1034,12 +1028,8 @@ print <<END
> > > >  </table>
> > > >  <hr size='1'>
> > > >  <table width='100%'>
> > > > -<tr><td class='base' colspan='4'><b>$Lang::tr{'advproxy redirector
> > > > children'}</b></td></tr>
> > > > -<tr><td class='base' >$Lang::tr{'processes'}:&nbsp;<img src='/blob.gif'
> > > > alt='*' /><input type='text' name='CHILDREN'
> > > > value='$proxysettings{'CHILDREN'}' size='5' /></td>
> > > >  END
> > > >  ;
> > > > -my $count = `ip n| wc -l`;
> > > > -if ( $count < 1 ){$count = 1;}
> > > >  if ( -e "/usr/bin/squidclamav" ) {
> > > >  	print "<td class='base'><b>".$Lang::tr{'advproxy
> > > > squidclamav'}."</b><br
> > > > />";
> > > >  	if ( ! -e "/var/run/clamav/clamd.pid" ){
> > > > @@ -1048,18 +1038,16 @@ if ( -e "/usr/bin/squidclamav" ) {
> > > >  		}
> > > >  	else {
> > > >  		print $Lang::tr{'advproxy enabled'}."<input
> > > > type='checkbox'
> > > > name='ENABLE_CLAMAV' ".$checked{'ENABLE_CLAMAV'}{'on'}." /><br />";
> > > > -		print "+ ".int(( $count**(1/3)) * 8);}
> > > > +}
> > > >  	print "</td>";
> > > >  } else {
> > > >  	print "<td></td>";
> > > >  }
> > > >  print "<td class='base'><a href='/cgi-
> > > > bin/urlfilter.cgi'><b>".$Lang::tr{'advproxy url filter'}."</a></b><br
> > > > />";
> > > >  print $Lang::tr{'advproxy enabled'}."<input type='checkbox'
> > > > name='ENABLE_FILTER' ".$checked{'ENABLE_FILTER'}{'on'}." /><br />";
> > > > -print "+ ".int(($count**(1/3)) * 6);
> > > >  print "</td>";
> > > >  print "<td class='base'><a href='/cgi-
> > > > bin/updatexlrator.cgi'><b>".$Lang::tr{'advproxy update
> > > > accelerator'}."</a></b><br />";
> > > >  print $Lang::tr{'advproxy enabled'}."<input type='checkbox'
> > > > name='ENABLE_UPDXLRATOR' ".$checked{'ENABLE_UPDXLRATOR'}{'on'}." /><br
> > > > />";
> > > > -print "+ ".int(($count**(1/3)) * 5);
> > > >  print "</td></tr>";
> > > >  print <<END
> > > >  </table>
> > > > @@ -4095,7 +4083,7 @@ END
> > > >  	if (($proxysettings{'ENABLE_FILTER'} eq 'on') ||
> > > > ($proxysettings{'ENABLE_UPDXLRATOR'} eq 'on') ||
> > > > ($proxysettings{'ENABLE_CLAMAV'} eq 'on'))
> > > >  	{
> > > >  		print FILE "url_rewrite_program
> > > > /usr/sbin/redirect_wrapper\n";
> > > > -		print FILE "url_rewrite_children
> > > > $proxysettings{'CHILDREN'}\n\n";
> > > > +		print FILE "url_rewrite_children ",
> > > > &General::number_cpu_cores(), "\n\n";
> > > >  	}
> > > >  
> > > >  	# Include file with user defined settings.
> > > > 
> > > 
> > > 
> > 
> > 
> 
> 

Re: [PATCH] BUG 11786 - squid: Remove setting for filter processes the number of Squid processes

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 8005 bytes --]

On 30.10.2018 17:05, Michael Tremer wrote:
> On Tue, 2018-10-30 at 15:37 +0100, Matthias Fischer wrote:
>> On 30.10.2018 15:03, Michael Tremer wrote:
>> > Hey,
>> 
>> Hello,
>> 
>> > do you have any idea why the redirector is busy?
>> 
>> Sorry: no. Long time ago, I searched for an optimal value but didn't
>> find any useful links. Right now, I'm testing 'squid 4.4" and its always
>> the same. Too few redirectors => warning.
>> 
>> I only found something like "If you see this warning, you should
>> increase the number of child processes and restart (or reconfigure)
>> Squid. If the queue size becomes twice the number of redirectors, Squid
>> aborts with a fatal message."
> 
> The last bit is not very good. Why does squid not queue the requests long
> enough? That means even CPU cores * 2 or 4 won't be enough. On a single-core
> machine that would only be 2 requests. LOL
> 
> Looking at the options that we have (
> http://www.squid-cache.org/Doc/config/url_rewrite_children/), could you please
> test this:
> 
>   url_rewrite_children X startup=N idle=N queue-size=64
> 
> With N being the number of cores that your system has. I do not think that there
> is any point in trying to run more than N queries at a time.
> 
> But squid doesn't start anything when it starts up. startup= changes that and
> idle= makes sure that all processes remain active.

Yep. I had this already in my 'proxy.cgi'.

I applied the patches again and changed the 'url_rewrite_children'-line
in 'proxy.cgi' to read:

print FILE "url_rewrite_children ", &General::number_cpu_cores(),"
startup=2 idle=2 queue-size=64\n\n";

This leads to this line in 'squid.conf':
url_rewrite_children 2 startup=2 idle=2 queue-size=64

No warnings so far. Testing ...

Best,
Matthias

> A queue size of 64 seems sensible to me because that is probably what the
> average tabloid newspaper page has. Maybe it should be 16 * N or so...
> 
> -Michael
> 
>> 
>> Taken from:
>> 
> http://etutorials.org/Server+Administration/Squid.+The+definitive+guide/Chapter+11.+Redirectors/11.4+Configuring+Squid/
>> 
>> So my current "40"-value was found through "try and look if the warning
>> comes again"...
>> 
>> Best,
>> Matthias
>> 
>> 
>> > -Michael
>> > 
>> > On Tue, 2018-10-30 at 14:54 +0100, Matthias Fischer wrote:
>> > > Hi,
>> > > 
>> > > I tested this patch today and I fear the resulting value is too low.
>> > > 
>> > > For me, its leads to:
>> > > 
>> > > ***SNIP***
>> > > ...
>> > > 2018/10/30 14:42:43 kid1| WARNING: All 2/2 redirector processes are busy.
>> > > 2018/10/30 14:42:43 kid1| WARNING: 4 pending requests queued
>> > > 2018/10/30 14:42:43 kid1| WARNING: Consider increasing the number of
>> > > redirector processes in your config file.
>> > > ...
>> > > 
>> > > Current setting is "40" (Duo box, 2 Cores, 'squidclamav, squidguard)
>> > > 
>> > > Best,
>> > > Matthias
>> > > 
>> > > On 30.10.2018 12:06, Daniel Weismüller wrote:
>> > > > I added a function to determine the number of cores.
>> > > > Now the number of squid processes will be equal to the number of logical
>> > > > cores.
>> > > > Further I removed the possibility of changing the number
>> > > > of squid processes in the proxy.cgi
>> > > > 
>> > > > Signed-off-by: Daniel Weismüller <daniel.weismueller(a)ipfire.org>
>> > > > ---
>> > > >  config/cfgroot/general-functions.pl |  7 +++++++
>> > > >  html/cgi-bin/proxy.cgi              | 16 ++--------------
>> > > >  2 files changed, 9 insertions(+), 14 deletions(-)
>> > > > 
>> > > > diff --git a/config/cfgroot/general-functions.pl
>> > > > b/config/cfgroot/general-
>> > > > functions.pl
>> > > > index 0577afe28..e8495e885 100644
>> > > > --- a/config/cfgroot/general-functions.pl
>> > > > +++ b/config/cfgroot/general-functions.pl
>> > > > @@ -1165,5 +1165,12 @@ sub dnssec_status() {
>> > > >  
>> > > >  	return $status;
>> > > >  }
>> > > > +sub number_cpu_cores() {
>> > > > +	open my $cpuinfo, "/proc/cpuinfo" or die "Can't open cpuinfo:
>> > > > $!\n";
>> > > > +	my $cores = scalar (map /^processor/, <$cpuinfo>);
>> > > > +	close $cpuinfo;
>> > > > +
>> > > > +	return $cores;
>> > > > +}
>> > > >  
>> > > >  1;
>> > > > diff --git a/html/cgi-bin/proxy.cgi b/html/cgi-bin/proxy.cgi
>> > > > index 738425b9a..92bebfe18 100644
>> > > > --- a/html/cgi-bin/proxy.cgi
>> > > > +++ b/html/cgi-bin/proxy.cgi
>> > > > @@ -287,7 +287,6 @@ $proxysettings{'IDENT_USER_ACL'} = 'positive';
>> > > >  $proxysettings{'ENABLE_FILTER'} = 'off';
>> > > >  $proxysettings{'ENABLE_UPDXLRATOR'} = 'off';
>> > > >  $proxysettings{'ENABLE_CLAMAV'} = 'off';
>> > > > -$proxysettings{'CHILDREN'} = '10';
>> > > >  
>> > > >  $ncsa_buttontext = $Lang::tr{'advproxy NCSA create user'};
>> > > >  
>> > > > @@ -437,11 +436,6 @@ if (($proxysettings{'ACTION'} eq $Lang::tr{'save'})
>> > > > ||
>> > > > ($proxysettings{'ACTION'}
>> > > >  		$errormessage = $Lang::tr{'invalid maximum incoming
>> > > > size'};
>> > > >  		goto ERROR;
>> > > >  	}
>> > > > -		if (!($proxysettings{'CHILDREN'} =~ /^\d+$/) ||
>> > > > ($proxysettings{'CHILDREN'} < 1))
>> > > > -	{
>> > > > -		$errormessage = $Lang::tr{'advproxy invalid num of
>> > > > children'};
>> > > > -		goto ERROR;
>> > > > -	}
>> > > >  	if ($proxysettings{'ENABLE_BROWSER_CHECK'} eq 'on')
>> > > >  	{
>> > > >  		$browser_regexp = '';
>> > > > @@ -1034,12 +1028,8 @@ print <<END
>> > > >  </table>
>> > > >  <hr size='1'>
>> > > >  <table width='100%'>
>> > > > -<tr><td class='base' colspan='4'><b>$Lang::tr{'advproxy redirector
>> > > > children'}</b></td></tr>
>> > > > -<tr><td class='base' >$Lang::tr{'processes'}:&nbsp;<img src='/blob.gif'
>> > > > alt='*' /><input type='text' name='CHILDREN'
>> > > > value='$proxysettings{'CHILDREN'}' size='5' /></td>
>> > > >  END
>> > > >  ;
>> > > > -my $count = `ip n| wc -l`;
>> > > > -if ( $count < 1 ){$count = 1;}
>> > > >  if ( -e "/usr/bin/squidclamav" ) {
>> > > >  	print "<td class='base'><b>".$Lang::tr{'advproxy
>> > > > squidclamav'}."</b><br
>> > > > />";
>> > > >  	if ( ! -e "/var/run/clamav/clamd.pid" ){
>> > > > @@ -1048,18 +1038,16 @@ if ( -e "/usr/bin/squidclamav" ) {
>> > > >  		}
>> > > >  	else {
>> > > >  		print $Lang::tr{'advproxy enabled'}."<input
>> > > > type='checkbox'
>> > > > name='ENABLE_CLAMAV' ".$checked{'ENABLE_CLAMAV'}{'on'}." /><br />";
>> > > > -		print "+ ".int(( $count**(1/3)) * 8);}
>> > > > +}
>> > > >  	print "</td>";
>> > > >  } else {
>> > > >  	print "<td></td>";
>> > > >  }
>> > > >  print "<td class='base'><a href='/cgi-
>> > > > bin/urlfilter.cgi'><b>".$Lang::tr{'advproxy url filter'}."</a></b><br
>> > > > />";
>> > > >  print $Lang::tr{'advproxy enabled'}."<input type='checkbox'
>> > > > name='ENABLE_FILTER' ".$checked{'ENABLE_FILTER'}{'on'}." /><br />";
>> > > > -print "+ ".int(($count**(1/3)) * 6);
>> > > >  print "</td>";
>> > > >  print "<td class='base'><a href='/cgi-
>> > > > bin/updatexlrator.cgi'><b>".$Lang::tr{'advproxy update
>> > > > accelerator'}."</a></b><br />";
>> > > >  print $Lang::tr{'advproxy enabled'}."<input type='checkbox'
>> > > > name='ENABLE_UPDXLRATOR' ".$checked{'ENABLE_UPDXLRATOR'}{'on'}." /><br
>> > > > />";
>> > > > -print "+ ".int(($count**(1/3)) * 5);
>> > > >  print "</td></tr>";
>> > > >  print <<END
>> > > >  </table>
>> > > > @@ -4095,7 +4083,7 @@ END
>> > > >  	if (($proxysettings{'ENABLE_FILTER'} eq 'on') ||
>> > > > ($proxysettings{'ENABLE_UPDXLRATOR'} eq 'on') ||
>> > > > ($proxysettings{'ENABLE_CLAMAV'} eq 'on'))
>> > > >  	{
>> > > >  		print FILE "url_rewrite_program
>> > > > /usr/sbin/redirect_wrapper\n";
>> > > > -		print FILE "url_rewrite_children
>> > > > $proxysettings{'CHILDREN'}\n\n";
>> > > > +		print FILE "url_rewrite_children ",
>> > > > &General::number_cpu_cores(), "\n\n";
>> > > >  	}
>> > > >  
>> > > >  	# Include file with user defined settings.

Re: [PATCH] BUG 11786 - squid: Remove setting for filter processes the number of Squid processes

[Daniel Weismüller]

[-- Attachment #1: Type: text/plain, Size: 9169 bytes --]

Hi Matthias

I've tested your changes. But for me it didn't work.
14:24:48	squid: 	Bungled /etc/squid/squid.conf line 136: url_rewrite_children 8 startup=2 idle=2 queue-size=64

According to the squid pages the "queue-size" argument isn't availlable for squid 3.5.
http://www.squid-cache.org/Versions/v3/3.5/cfgman/url_rewrite_children.html

I changed the code to: print FILE "url_rewrite_children ",&General::number_cpu_cores()," startup=",&General::number_cpu_cores()," idle=",&General::number_cpu_cores()," \n\n";

This seems to work with squid 3.5 and leads to this line in 'squid.conf': url_rewrite_children 8 startup=8 idle=8

As you can see, my Test-ipfire has 8 virtual cores, so I couldn't see the above problems.
Thanks for testing it!

-
Daniel


Matthias Fischer – Tue, 30. October 2018 18:12
> On 30.10.2018 17:05, Michael Tremer wrote:
> > On Tue, 2018-10-30 at 15:37 +0100, Matthias Fischer wrote:
> >> On 30.10.2018 15:03, Michael Tremer wrote:
> >> > Hey,
> >> 
> >> Hello,
> >> 
> >> > do you have any idea why the redirector is busy?
> >> 
> >> Sorry: no. Long time ago, I searched for an optimal value but didn't
> >> find any useful links. Right now, I'm testing 'squid 4.4" and its always
> >> the same. Too few redirectors => warning.
> >> 
> >> I only found something like "If you see this warning, you should
> >> increase the number of child processes and restart (or reconfigure)
> >> Squid. If the queue size becomes twice the number of redirectors, Squid
> >> aborts with a fatal message."
> > 
> > The last bit is not very good. Why does squid not queue the requests long
> > enough? That means even CPU cores * 2 or 4 won't be enough. On a single-core
> > machine that would only be 2 requests. LOL
> > 
> > Looking at the options that we have (
> > www.squid-cache.org/Doc/config/url_rewrite_children/), could you please
> > test this:
> > 
> > url_rewrite_children X startup=N idle=N queue-size=64
> > 
> > With N being the number of cores that your system has. I do not think that
> there
> > is any point in trying to run more than N queries at a time.
> > 
> > But squid doesn't start anything when it starts up. startup= changes that
> and
> > idle= makes sure that all processes remain active.
> 
> Yep. I had this already in my 'proxy.cgi'.
> 
> I applied the patches again and changed the 'url_rewrite_children'-line
> in 'proxy.cgi' to read:
> 
> print FILE "url_rewrite_children ", &General::number_cpu_cores(),"
> startup=2 idle=2 queue-size=64\n\n";
> 
> This leads to this line in 'squid.conf':
> url_rewrite_children 2 startup=2 idle=2 queue-size=64
> 
> No warnings so far. Testing ...
> 
> Best,
> Matthias
> 
> > A queue size of 64 seems sensible to me because that is probably what the
> > average tabloid newspaper page has. Maybe it should be 16 * N or so...
> > 
> > -Michael
> > 
> >> 
> >> Taken from:
> >> 
> > 
> etutorials.org/Server+Administration/Squid.+The+definitive+guide/Chapter+11.+Redirectors/11.4+Configuring+Squid/
> >> 
> >> So my current "40"-value was found through "try and look if the warning
> >> comes again"...
> >> 
> >> Best,
> >> Matthias
> >> 
> >> 
> >> > -Michael
> >> > 
> >> > On Tue, 2018-10-30 at 14:54 +0100, Matthias Fischer wrote:
> >> > > Hi,
> >> > > 
> >> > > I tested this patch today and I fear the resulting value is too low.
> >> > > 
> >> > > For me, its leads to:
> >> > > 
> >> > > ***SNIP***
> >> > > ...
> >> > > 2018/10/30 14:42:43 kid1| WARNING: All 2/2 redirector processes are
> busy.
> >> > > 2018/10/30 14:42:43 kid1| WARNING: 4 pending requests queued
> >> > > 2018/10/30 14:42:43 kid1| WARNING: Consider increasing the number of
> >> > > redirector processes in your config file.
> >> > > ...
> >> > > 
> >> > > Current setting is "40" (Duo box, 2 Cores, 'squidclamav, squidguard)
> >> > > 
> >> > > Best,
> >> > > Matthias
> >> > > 
> >> > > On 30.10.2018 12:06, Daniel Weismüller wrote:
> >> > > > I added a function to determine the number of cores.
> >> > > > Now the number of squid processes will be equal to the number of
> logical
> >> > > > cores.
> >> > > > Further I removed the possibility of changing the number
> >> > > > of squid processes in the proxy.cgi
> >> > > > 
> >> > > > Signed-off-by: Daniel Weismüller <daniel.weismueller(a)ipfire.org>
> >> > > > ---
> >> > > > config/cfgroot/general-functions.pl | 7 +++++++
> >> > > > html/cgi-bin/proxy.cgi | 16 ++--------------
> >> > > > 2 files changed, 9 insertions(+), 14 deletions(-)
> >> > > > 
> >> > > > diff --git a/config/cfgroot/general-functions.pl
> >> > > > b/config/cfgroot/general-
> >> > > > functions.pl
> >> > > > index 0577afe28..e8495e885 100644
> >> > > > --- a/config/cfgroot/general-functions.pl
> >> > > > +++ b/config/cfgroot/general-functions.pl
> >> > > > @@ -1165,5 +1165,12 @@ sub dnssec_status() {
> >> > > > 
> >> > > > return $status;
> >> > > > }
> >> > > > +sub number_cpu_cores() {
> >> > > > + open my $cpuinfo, "/proc/cpuinfo" or die "Can't open cpuinfo:
> >> > > > $!\n";
> >> > > > + my $cores = scalar (map /^processor/, <$cpuinfo>);
> >> > > > + close $cpuinfo;
> >> > > > +
> >> > > > + return $cores;
> >> > > > +}
> >> > > > 
> >> > > > 1;
> >> > > > diff --git a/html/cgi-bin/proxy.cgi b/html/cgi-bin/proxy.cgi
> >> > > > index 738425b9a..92bebfe18 100644
> >> > > > --- a/html/cgi-bin/proxy.cgi
> >> > > > +++ b/html/cgi-bin/proxy.cgi
> >> > > > @@ -287,7 +287,6 @@ $proxysettings{'IDENT_USER_ACL'} = 'positive';
> >> > > > $proxysettings{'ENABLE_FILTER'} = 'off';
> >> > > > $proxysettings{'ENABLE_UPDXLRATOR'} = 'off';
> >> > > > $proxysettings{'ENABLE_CLAMAV'} = 'off';
> >> > > > -$proxysettings{'CHILDREN'} = '10';
> >> > > > 
> >> > > > $ncsa_buttontext = $Lang::tr{'advproxy NCSA create user'};
> >> > > > 
> >> > > > @@ -437,11 +436,6 @@ if (($proxysettings{'ACTION'} eq
> $Lang::tr{'save'})
> >> > > > ||
> >> > > > ($proxysettings{'ACTION'}
> >> > > > $errormessage = $Lang::tr{'invalid maximum incoming
> >> > > > size'};
> >> > > > goto ERROR;
> >> > > > }
> >> > > > - if (!($proxysettings{'CHILDREN'} =~ /^\d+$/) ||
> >> > > > ($proxysettings{'CHILDREN'} < 1))
> >> > > > - {
> >> > > > - $errormessage = $Lang::tr{'advproxy invalid num of
> >> > > > children'};
> >> > > > - goto ERROR;
> >> > > > - }
> >> > > > if ($proxysettings{'ENABLE_BROWSER_CHECK'} eq 'on')
> >> > > > {
> >> > > > $browser_regexp = '';
> >> > > > @@ -1034,12 +1028,8 @@ print <<END
> >> > > > </table>
> >> > > > <hr size='1'>
> >> > > > <table width='100%'>
> >> > > > -<tr><td class='base' colspan='4'><b>$Lang::tr{'advproxy redirector
> >> > > > children'}</b></td></tr>
> >> > > > -<tr><td class='base' >$Lang::tr{'processes'}:&nbsp;<img
> src='/blob.gif'
> >> > > > alt='*' /><input type='text' name='CHILDREN'
> >> > > > value='$proxysettings{'CHILDREN'}' size='5' /></td>
> >> > > > END
> >> > > > ;
> >> > > > -my $count = `ip n| wc -l`;
> >> > > > -if ( $count < 1 ){$count = 1;}
> >> > > > if ( -e "/usr/bin/squidclamav" ) {
> >> > > > print "<td class='base'><b>".$Lang::tr{'advproxy
> >> > > > squidclamav'}."</b><br
> >> > > > />";
> >> > > > if ( ! -e "/var/run/clamav/clamd.pid" ){
> >> > > > @@ -1048,18 +1038,16 @@ if ( -e "/usr/bin/squidclamav" ) {
> >> > > > }
> >> > > > else {
> >> > > > print $Lang::tr{'advproxy enabled'}."<input
> >> > > > type='checkbox'
> >> > > > name='ENABLE_CLAMAV' ".$checked{'ENABLE_CLAMAV'}{'on'}." /><br />";
> >> > > > - print "+ ".int(( $count**(1/3)) * 8);}
> >> > > > +}
> >> > > > print "</td>";
> >> > > > } else {
> >> > > > print "<td></td>";
> >> > > > }
> >> > > > print "<td class='base'><a href='/cgi-
> >> > > > bin/urlfilter.cgi'><b>".$Lang::tr{'advproxy url filter'}."</a></b><br
> >> > > > />";
> >> > > > print $Lang::tr{'advproxy enabled'}."<input type='checkbox'
> >> > > > name='ENABLE_FILTER' ".$checked{'ENABLE_FILTER'}{'on'}." /><br />";
> >> > > > -print "+ ".int(($count**(1/3)) * 6);
> >> > > > print "</td>";
> >> > > > print "<td class='base'><a href='/cgi-
> >> > > > bin/updatexlrator.cgi'><b>".$Lang::tr{'advproxy update
> >> > > > accelerator'}."</a></b><br />";
> >> > > > print $Lang::tr{'advproxy enabled'}."<input type='checkbox'
> >> > > > name='ENABLE_UPDXLRATOR' ".$checked{'ENABLE_UPDXLRATOR'}{'on'}."
> /><br
> >> > > > />";
> >> > > > -print "+ ".int(($count**(1/3)) * 5);
> >> > > > print "</td></tr>";
> >> > > > print <<END
> >> > > > </table>
> >> > > > @@ -4095,7 +4083,7 @@ END
> >> > > > if (($proxysettings{'ENABLE_FILTER'} eq 'on') ||
> >> > > > ($proxysettings{'ENABLE_UPDXLRATOR'} eq 'on') ||
> >> > > > ($proxysettings{'ENABLE_CLAMAV'} eq 'on'))
> >> > > > {
> >> > > > print FILE "url_rewrite_program
> >> > > > /usr/sbin/redirect_wrapper\n";
> >> > > > - print FILE "url_rewrite_children
> >> > > > $proxysettings{'CHILDREN'}\n\n";
> >> > > > + print FILE "url_rewrite_children ",
> >> > > > &General::number_cpu_cores(), "\n\n";
> >> > > > }
> >> > > > 
> >> > > > # Include file with user defined settings.

Re: [PATCH] BUG 11786 - squid: Remove setting for filter processes the number of Squid processes

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 10366 bytes --]

On Fri, 2018-11-02 at 14:13 +0000, Daniel Weismüller wrote:
> Hi Matthias
> 
> I've tested your changes. But for me it didn't work.
> 14:24:48	squid: 	Bungled /etc/squid/squid.conf line 136:
> url_rewrite_children 8 startup=2 idle=2 queue-size=64
> 
> According to the squid pages the "queue-size" argument isn't availlable for
> squid 3.5.
> http://www.squid-cache.org/Versions/v3/3.5/cfgman/url_rewrite_children.html
> 
> I changed the code to: print FILE "url_rewrite_children
> ",&General::number_cpu_cores()," startup=",&General::number_cpu_cores(),"
> idle=",&General::number_cpu_cores()," \n\n";
> 
> This seems to work with squid 3.5 and leads to this line in 'squid.conf':
> url_rewrite_children 8 startup=8 idle=8

It is quite important to increase the size of the queue, because in your case 16
might be enough, but that isn't true if that code is executed on a system with
only one core.

But I strongly agree with startup and idle being the number of cores too, so
that always a constant number of redirectors is running.

-Michael

> 
> As you can see, my Test-ipfire has 8 virtual cores, so I couldn't see the
> above problems.
> Thanks for testing it!
> 
> -
> Daniel
> 
> 
> Matthias Fischer – Tue, 30. October 2018 18:12
> > On 30.10.2018 17:05, Michael Tremer wrote:
> > > On Tue, 2018-10-30 at 15:37 +0100, Matthias Fischer wrote:
> > > > On 30.10.2018 15:03, Michael Tremer wrote:
> > > > > Hey,
> > > > 
> > > > Hello,
> > > > 
> > > > > do you have any idea why the redirector is busy?
> > > > 
> > > > Sorry: no. Long time ago, I searched for an optimal value but didn't
> > > > find any useful links. Right now, I'm testing 'squid 4.4" and its always
> > > > the same. Too few redirectors => warning.
> > > > 
> > > > I only found something like "If you see this warning, you should
> > > > increase the number of child processes and restart (or reconfigure)
> > > > Squid. If the queue size becomes twice the number of redirectors, Squid
> > > > aborts with a fatal message."
> > > 
> > > The last bit is not very good. Why does squid not queue the requests long
> > > enough? That means even CPU cores * 2 or 4 won't be enough. On a single-
> > > core
> > > machine that would only be 2 requests. LOL
> > > 
> > > Looking at the options that we have (
> > > www.squid-cache.org/Doc/config/url_rewrite_children/), could you please
> > > test this:
> > > 
> > > url_rewrite_children X startup=N idle=N queue-size=64
> > > 
> > > With N being the number of cores that your system has. I do not think that
> > 
> > there
> > > is any point in trying to run more than N queries at a time.
> > > 
> > > But squid doesn't start anything when it starts up. startup= changes that
> > 
> > and
> > > idle= makes sure that all processes remain active.
> > 
> > Yep. I had this already in my 'proxy.cgi'.
> > 
> > I applied the patches again and changed the 'url_rewrite_children'-line
> > in 'proxy.cgi' to read:
> > 
> > print FILE "url_rewrite_children ", &General::number_cpu_cores(),"
> > startup=2 idle=2 queue-size=64\n\n";
> > 
> > This leads to this line in 'squid.conf':
> > url_rewrite_children 2 startup=2 idle=2 queue-size=64
> > 
> > No warnings so far. Testing ...
> > 
> > Best,
> > Matthias
> > 
> > > A queue size of 64 seems sensible to me because that is probably what the
> > > average tabloid newspaper page has. Maybe it should be 16 * N or so...
> > > 
> > > -Michael
> > > 
> > > > 
> > > > Taken from:
> > > > 
> > 
> > etutorials.org/Server+Administration/Squid.+The+definitive+guide/Chapter+11.
> > +Redirectors/11.4+Configuring+Squid/
> > > > 
> > > > So my current "40"-value was found through "try and look if the warning
> > > > comes again"...
> > > > 
> > > > Best,
> > > > Matthias
> > > > 
> > > > 
> > > > > -Michael
> > > > > 
> > > > > On Tue, 2018-10-30 at 14:54 +0100, Matthias Fischer wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > I tested this patch today and I fear the resulting value is too low.
> > > > > > 
> > > > > > For me, its leads to:
> > > > > > 
> > > > > > ***SNIP***
> > > > > > ...
> > > > > > 2018/10/30 14:42:43 kid1| WARNING: All 2/2 redirector processes are
> > 
> > busy.
> > > > > > 2018/10/30 14:42:43 kid1| WARNING: 4 pending requests queued
> > > > > > 2018/10/30 14:42:43 kid1| WARNING: Consider increasing the number of
> > > > > > redirector processes in your config file.
> > > > > > ...
> > > > > > 
> > > > > > Current setting is "40" (Duo box, 2 Cores, 'squidclamav, squidguard)
> > > > > > 
> > > > > > Best,
> > > > > > Matthias
> > > > > > 
> > > > > > On 30.10.2018 12:06, Daniel Weismüller wrote:
> > > > > > > I added a function to determine the number of cores.
> > > > > > > Now the number of squid processes will be equal to the number of
> > 
> > logical
> > > > > > > cores.
> > > > > > > Further I removed the possibility of changing the number
> > > > > > > of squid processes in the proxy.cgi
> > > > > > > 
> > > > > > > Signed-off-by: Daniel Weismüller <daniel.weismueller(a)ipfire.org>
> > > > > > > ---
> > > > > > > config/cfgroot/general-functions.pl | 7 +++++++
> > > > > > > html/cgi-bin/proxy.cgi | 16 ++--------------
> > > > > > > 2 files changed, 9 insertions(+), 14 deletions(-)
> > > > > > > 
> > > > > > > diff --git a/config/cfgroot/general-functions.pl
> > > > > > > b/config/cfgroot/general-
> > > > > > > functions.pl
> > > > > > > index 0577afe28..e8495e885 100644
> > > > > > > --- a/config/cfgroot/general-functions.pl
> > > > > > > +++ b/config/cfgroot/general-functions.pl
> > > > > > > @@ -1165,5 +1165,12 @@ sub dnssec_status() {
> > > > > > > 
> > > > > > > return $status;
> > > > > > > }
> > > > > > > +sub number_cpu_cores() {
> > > > > > > + open my $cpuinfo, "/proc/cpuinfo" or die "Can't open cpuinfo:
> > > > > > > $!\n";
> > > > > > > + my $cores = scalar (map /^processor/, <$cpuinfo>);
> > > > > > > + close $cpuinfo;
> > > > > > > +
> > > > > > > + return $cores;
> > > > > > > +}
> > > > > > > 
> > > > > > > 1;
> > > > > > > diff --git a/html/cgi-bin/proxy.cgi b/html/cgi-bin/proxy.cgi
> > > > > > > index 738425b9a..92bebfe18 100644
> > > > > > > --- a/html/cgi-bin/proxy.cgi
> > > > > > > +++ b/html/cgi-bin/proxy.cgi
> > > > > > > @@ -287,7 +287,6 @@ $proxysettings{'IDENT_USER_ACL'} = 'positive';
> > > > > > > $proxysettings{'ENABLE_FILTER'} = 'off';
> > > > > > > $proxysettings{'ENABLE_UPDXLRATOR'} = 'off';
> > > > > > > $proxysettings{'ENABLE_CLAMAV'} = 'off';
> > > > > > > -$proxysettings{'CHILDREN'} = '10';
> > > > > > > 
> > > > > > > $ncsa_buttontext = $Lang::tr{'advproxy NCSA create user'};
> > > > > > > 
> > > > > > > @@ -437,11 +436,6 @@ if (($proxysettings{'ACTION'} eq
> > 
> > $Lang::tr{'save'})
> > > > > > > > > 
> > > > > > > 
> > > > > > > ($proxysettings{'ACTION'}
> > > > > > > $errormessage = $Lang::tr{'invalid maximum incoming
> > > > > > > size'};
> > > > > > > goto ERROR;
> > > > > > > }
> > > > > > > - if (!($proxysettings{'CHILDREN'} =~ /^\d+$/) ||
> > > > > > > ($proxysettings{'CHILDREN'} < 1))
> > > > > > > - {
> > > > > > > - $errormessage = $Lang::tr{'advproxy invalid num of
> > > > > > > children'};
> > > > > > > - goto ERROR;
> > > > > > > - }
> > > > > > > if ($proxysettings{'ENABLE_BROWSER_CHECK'} eq 'on')
> > > > > > > {
> > > > > > > $browser_regexp = '';
> > > > > > > @@ -1034,12 +1028,8 @@ print <<END
> > > > > > > </table>
> > > > > > > <hr size='1'>
> > > > > > > <table width='100%'>
> > > > > > > -<tr><td class='base' colspan='4'><b>$Lang::tr{'advproxy
> > > > > > > redirector
> > > > > > > children'}</b></td></tr>
> > > > > > > -<tr><td class='base' >$Lang::tr{'processes'}:&nbsp;<img
> > 
> > src='/blob.gif'
> > > > > > > alt='*' /><input type='text' name='CHILDREN'
> > > > > > > value='$proxysettings{'CHILDREN'}' size='5' /></td>
> > > > > > > END
> > > > > > > ;
> > > > > > > -my $count = `ip n| wc -l`;
> > > > > > > -if ( $count < 1 ){$count = 1;}
> > > > > > > if ( -e "/usr/bin/squidclamav" ) {
> > > > > > > print "<td class='base'><b>".$Lang::tr{'advproxy
> > > > > > > squidclamav'}."</b><br
> > > > > > > />";
> > > > > > > if ( ! -e "/var/run/clamav/clamd.pid" ){
> > > > > > > @@ -1048,18 +1038,16 @@ if ( -e "/usr/bin/squidclamav" ) {
> > > > > > > }
> > > > > > > else {
> > > > > > > print $Lang::tr{'advproxy enabled'}."<input
> > > > > > > type='checkbox'
> > > > > > > name='ENABLE_CLAMAV' ".$checked{'ENABLE_CLAMAV'}{'on'}." /><br
> > > > > > > />";
> > > > > > > - print "+ ".int(( $count**(1/3)) * 8);}
> > > > > > > +}
> > > > > > > print "</td>";
> > > > > > > } else {
> > > > > > > print "<td></td>";
> > > > > > > }
> > > > > > > print "<td class='base'><a href='/cgi-
> > > > > > > bin/urlfilter.cgi'><b>".$Lang::tr{'advproxy url
> > > > > > > filter'}."</a></b><br
> > > > > > > />";
> > > > > > > print $Lang::tr{'advproxy enabled'}."<input type='checkbox'
> > > > > > > name='ENABLE_FILTER' ".$checked{'ENABLE_FILTER'}{'on'}." /><br
> > > > > > > />";
> > > > > > > -print "+ ".int(($count**(1/3)) * 6);
> > > > > > > print "</td>";
> > > > > > > print "<td class='base'><a href='/cgi-
> > > > > > > bin/updatexlrator.cgi'><b>".$Lang::tr{'advproxy update
> > > > > > > accelerator'}."</a></b><br />";
> > > > > > > print $Lang::tr{'advproxy enabled'}."<input type='checkbox'
> > > > > > > name='ENABLE_UPDXLRATOR' ".$checked{'ENABLE_UPDXLRATOR'}{'on'}."
> > 
> > /><br
> > > > > > > />";
> > > > > > > -print "+ ".int(($count**(1/3)) * 5);
> > > > > > > print "</td></tr>";
> > > > > > > print <<END
> > > > > > > </table>
> > > > > > > @@ -4095,7 +4083,7 @@ END
> > > > > > > if (($proxysettings{'ENABLE_FILTER'} eq 'on') ||
> > > > > > > ($proxysettings{'ENABLE_UPDXLRATOR'} eq 'on') ||
> > > > > > > ($proxysettings{'ENABLE_CLAMAV'} eq 'on'))
> > > > > > > {
> > > > > > > print FILE "url_rewrite_program
> > > > > > > /usr/sbin/redirect_wrapper\n";
> > > > > > > - print FILE "url_rewrite_children
> > > > > > > $proxysettings{'CHILDREN'}\n\n";
> > > > > > > + print FILE "url_rewrite_children ",
> > > > > > > &General::number_cpu_cores(), "\n\n";
> > > > > > > }
> > > > > > > 
> > > > > > > # Include file with user defined settings.

Re: [PATCH] BUG 11786 - squid: Remove setting for filter processes the number of Squid processes

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 11160 bytes --]

Hi,

On 02.11.2018 18:07, Michael Tremer wrote:
> On Fri, 2018-11-02 at 14:13 +0000, Daniel Weismüller wrote:
>> Hi Matthias
>> 
>> I've tested your changes. But for me it didn't work.
>> 14:24:48	squid: 	Bungled /etc/squid/squid.conf line 136:
>> url_rewrite_children 8 startup=2 idle=2 queue-size=64
>> 
>> According to the squid pages the "queue-size" argument isn't availlable for
>> squid 3.5.
>> http://www.squid-cache.org/Versions/v3/3.5/cfgman/url_rewrite_children.html

Ups. Sorry, my fault! I didn't check this and my production machine runs
'squid 4.4'.

>> I changed the code to: print FILE "url_rewrite_children
>> ",&General::number_cpu_cores()," startup=",&General::number_cpu_cores(),"
>> idle=",&General::number_cpu_cores()," \n\n";
>> 
>> This seems to work with squid 3.5 and leads to this line in 'squid.conf':
>> url_rewrite_children 8 startup=8 idle=8

As far as I remember, this could me the bare minimum for 'squid 3.x'.

> It is quite important to increase the size of the queue, because in your case 16
> might be enough, but that isn't true if that code is executed on a system with
> only one core.
> 
> But I strongly agree with startup and idle being the number of cores too, so
> that always a constant number of redirectors is running.

FYI:
I've been using 'squid 4.4' now for over a year. Meanwhile this version
has been declared stable beside '3.5.28':

=> http://www.squid-cache.org/Versions/

Any thoughts of upgrading?

> -Michael
> 
>> 
>> As you can see, my Test-ipfire has 8 virtual cores, so I couldn't see the
>> above problems.

Any warnings in '/var/log/squid/cache.log'?

>> Thanks for testing it!

No problem -you're welcome! ;-)

Best,
Matthias

>> -
>> Daniel
>> 
>> 
>> Matthias Fischer – Tue, 30. October 2018 18:12
>> > On 30.10.2018 17:05, Michael Tremer wrote:
>> > > On Tue, 2018-10-30 at 15:37 +0100, Matthias Fischer wrote:
>> > > > On 30.10.2018 15:03, Michael Tremer wrote:
>> > > > > Hey,
>> > > > 
>> > > > Hello,
>> > > > 
>> > > > > do you have any idea why the redirector is busy?
>> > > > 
>> > > > Sorry: no. Long time ago, I searched for an optimal value but didn't
>> > > > find any useful links. Right now, I'm testing 'squid 4.4" and its always
>> > > > the same. Too few redirectors => warning.
>> > > > 
>> > > > I only found something like "If you see this warning, you should
>> > > > increase the number of child processes and restart (or reconfigure)
>> > > > Squid. If the queue size becomes twice the number of redirectors, Squid
>> > > > aborts with a fatal message."
>> > > 
>> > > The last bit is not very good. Why does squid not queue the requests long
>> > > enough? That means even CPU cores * 2 or 4 won't be enough. On a single-
>> > > core
>> > > machine that would only be 2 requests. LOL
>> > > 
>> > > Looking at the options that we have (
>> > > www.squid-cache.org/Doc/config/url_rewrite_children/), could you please
>> > > test this:
>> > > 
>> > > url_rewrite_children X startup=N idle=N queue-size=64
>> > > 
>> > > With N being the number of cores that your system has. I do not think that
>> > 
>> > there
>> > > is any point in trying to run more than N queries at a time.
>> > > 
>> > > But squid doesn't start anything when it starts up. startup= changes that
>> > 
>> > and
>> > > idle= makes sure that all processes remain active.
>> > 
>> > Yep. I had this already in my 'proxy.cgi'.
>> > 
>> > I applied the patches again and changed the 'url_rewrite_children'-line
>> > in 'proxy.cgi' to read:
>> > 
>> > print FILE "url_rewrite_children ", &General::number_cpu_cores(),"
>> > startup=2 idle=2 queue-size=64\n\n";
>> > 
>> > This leads to this line in 'squid.conf':
>> > url_rewrite_children 2 startup=2 idle=2 queue-size=64
>> > 
>> > No warnings so far. Testing ...
>> > 
>> > Best,
>> > Matthias
>> > 
>> > > A queue size of 64 seems sensible to me because that is probably what the
>> > > average tabloid newspaper page has. Maybe it should be 16 * N or so...
>> > > 
>> > > -Michael
>> > > 
>> > > > 
>> > > > Taken from:
>> > > > 
>> > 
>> > etutorials.org/Server+Administration/Squid.+The+definitive+guide/Chapter+11.
>> > +Redirectors/11.4+Configuring+Squid/
>> > > > 
>> > > > So my current "40"-value was found through "try and look if the warning
>> > > > comes again"...
>> > > > 
>> > > > Best,
>> > > > Matthias
>> > > > 
>> > > > 
>> > > > > -Michael
>> > > > > 
>> > > > > On Tue, 2018-10-30 at 14:54 +0100, Matthias Fischer wrote:
>> > > > > > Hi,
>> > > > > > 
>> > > > > > I tested this patch today and I fear the resulting value is too low.
>> > > > > > 
>> > > > > > For me, its leads to:
>> > > > > > 
>> > > > > > ***SNIP***
>> > > > > > ...
>> > > > > > 2018/10/30 14:42:43 kid1| WARNING: All 2/2 redirector processes are
>> > 
>> > busy.
>> > > > > > 2018/10/30 14:42:43 kid1| WARNING: 4 pending requests queued
>> > > > > > 2018/10/30 14:42:43 kid1| WARNING: Consider increasing the number of
>> > > > > > redirector processes in your config file.
>> > > > > > ...
>> > > > > > 
>> > > > > > Current setting is "40" (Duo box, 2 Cores, 'squidclamav, squidguard)
>> > > > > > 
>> > > > > > Best,
>> > > > > > Matthias
>> > > > > > 
>> > > > > > On 30.10.2018 12:06, Daniel Weismüller wrote:
>> > > > > > > I added a function to determine the number of cores.
>> > > > > > > Now the number of squid processes will be equal to the number of
>> > 
>> > logical
>> > > > > > > cores.
>> > > > > > > Further I removed the possibility of changing the number
>> > > > > > > of squid processes in the proxy.cgi
>> > > > > > > 
>> > > > > > > Signed-off-by: Daniel Weismüller <daniel.weismueller(a)ipfire.org>
>> > > > > > > ---
>> > > > > > > config/cfgroot/general-functions.pl | 7 +++++++
>> > > > > > > html/cgi-bin/proxy.cgi | 16 ++--------------
>> > > > > > > 2 files changed, 9 insertions(+), 14 deletions(-)
>> > > > > > > 
>> > > > > > > diff --git a/config/cfgroot/general-functions.pl
>> > > > > > > b/config/cfgroot/general-
>> > > > > > > functions.pl
>> > > > > > > index 0577afe28..e8495e885 100644
>> > > > > > > --- a/config/cfgroot/general-functions.pl
>> > > > > > > +++ b/config/cfgroot/general-functions.pl
>> > > > > > > @@ -1165,5 +1165,12 @@ sub dnssec_status() {
>> > > > > > > 
>> > > > > > > return $status;
>> > > > > > > }
>> > > > > > > +sub number_cpu_cores() {
>> > > > > > > + open my $cpuinfo, "/proc/cpuinfo" or die "Can't open cpuinfo:
>> > > > > > > $!\n";
>> > > > > > > + my $cores = scalar (map /^processor/, <$cpuinfo>);
>> > > > > > > + close $cpuinfo;
>> > > > > > > +
>> > > > > > > + return $cores;
>> > > > > > > +}
>> > > > > > > 
>> > > > > > > 1;
>> > > > > > > diff --git a/html/cgi-bin/proxy.cgi b/html/cgi-bin/proxy.cgi
>> > > > > > > index 738425b9a..92bebfe18 100644
>> > > > > > > --- a/html/cgi-bin/proxy.cgi
>> > > > > > > +++ b/html/cgi-bin/proxy.cgi
>> > > > > > > @@ -287,7 +287,6 @@ $proxysettings{'IDENT_USER_ACL'} = 'positive';
>> > > > > > > $proxysettings{'ENABLE_FILTER'} = 'off';
>> > > > > > > $proxysettings{'ENABLE_UPDXLRATOR'} = 'off';
>> > > > > > > $proxysettings{'ENABLE_CLAMAV'} = 'off';
>> > > > > > > -$proxysettings{'CHILDREN'} = '10';
>> > > > > > > 
>> > > > > > > $ncsa_buttontext = $Lang::tr{'advproxy NCSA create user'};
>> > > > > > > 
>> > > > > > > @@ -437,11 +436,6 @@ if (($proxysettings{'ACTION'} eq
>> > 
>> > $Lang::tr{'save'})
>> > > > > > > > > 
>> > > > > > > 
>> > > > > > > ($proxysettings{'ACTION'}
>> > > > > > > $errormessage = $Lang::tr{'invalid maximum incoming
>> > > > > > > size'};
>> > > > > > > goto ERROR;
>> > > > > > > }
>> > > > > > > - if (!($proxysettings{'CHILDREN'} =~ /^\d+$/) ||
>> > > > > > > ($proxysettings{'CHILDREN'} < 1))
>> > > > > > > - {
>> > > > > > > - $errormessage = $Lang::tr{'advproxy invalid num of
>> > > > > > > children'};
>> > > > > > > - goto ERROR;
>> > > > > > > - }
>> > > > > > > if ($proxysettings{'ENABLE_BROWSER_CHECK'} eq 'on')
>> > > > > > > {
>> > > > > > > $browser_regexp = '';
>> > > > > > > @@ -1034,12 +1028,8 @@ print <<END
>> > > > > > > </table>
>> > > > > > > <hr size='1'>
>> > > > > > > <table width='100%'>
>> > > > > > > -<tr><td class='base' colspan='4'><b>$Lang::tr{'advproxy
>> > > > > > > redirector
>> > > > > > > children'}</b></td></tr>
>> > > > > > > -<tr><td class='base' >$Lang::tr{'processes'}:&nbsp;<img
>> > 
>> > src='/blob.gif'
>> > > > > > > alt='*' /><input type='text' name='CHILDREN'
>> > > > > > > value='$proxysettings{'CHILDREN'}' size='5' /></td>
>> > > > > > > END
>> > > > > > > ;
>> > > > > > > -my $count = `ip n| wc -l`;
>> > > > > > > -if ( $count < 1 ){$count = 1;}
>> > > > > > > if ( -e "/usr/bin/squidclamav" ) {
>> > > > > > > print "<td class='base'><b>".$Lang::tr{'advproxy
>> > > > > > > squidclamav'}."</b><br
>> > > > > > > />";
>> > > > > > > if ( ! -e "/var/run/clamav/clamd.pid" ){
>> > > > > > > @@ -1048,18 +1038,16 @@ if ( -e "/usr/bin/squidclamav" ) {
>> > > > > > > }
>> > > > > > > else {
>> > > > > > > print $Lang::tr{'advproxy enabled'}."<input
>> > > > > > > type='checkbox'
>> > > > > > > name='ENABLE_CLAMAV' ".$checked{'ENABLE_CLAMAV'}{'on'}." /><br
>> > > > > > > />";
>> > > > > > > - print "+ ".int(( $count**(1/3)) * 8);}
>> > > > > > > +}
>> > > > > > > print "</td>";
>> > > > > > > } else {
>> > > > > > > print "<td></td>";
>> > > > > > > }
>> > > > > > > print "<td class='base'><a href='/cgi-
>> > > > > > > bin/urlfilter.cgi'><b>".$Lang::tr{'advproxy url
>> > > > > > > filter'}."</a></b><br
>> > > > > > > />";
>> > > > > > > print $Lang::tr{'advproxy enabled'}."<input type='checkbox'
>> > > > > > > name='ENABLE_FILTER' ".$checked{'ENABLE_FILTER'}{'on'}." /><br
>> > > > > > > />";
>> > > > > > > -print "+ ".int(($count**(1/3)) * 6);
>> > > > > > > print "</td>";
>> > > > > > > print "<td class='base'><a href='/cgi-
>> > > > > > > bin/updatexlrator.cgi'><b>".$Lang::tr{'advproxy update
>> > > > > > > accelerator'}."</a></b><br />";
>> > > > > > > print $Lang::tr{'advproxy enabled'}."<input type='checkbox'
>> > > > > > > name='ENABLE_UPDXLRATOR' ".$checked{'ENABLE_UPDXLRATOR'}{'on'}."
>> > 
>> > /><br
>> > > > > > > />";
>> > > > > > > -print "+ ".int(($count**(1/3)) * 5);
>> > > > > > > print "</td></tr>";
>> > > > > > > print <<END
>> > > > > > > </table>
>> > > > > > > @@ -4095,7 +4083,7 @@ END
>> > > > > > > if (($proxysettings{'ENABLE_FILTER'} eq 'on') ||
>> > > > > > > ($proxysettings{'ENABLE_UPDXLRATOR'} eq 'on') ||
>> > > > > > > ($proxysettings{'ENABLE_CLAMAV'} eq 'on'))
>> > > > > > > {
>> > > > > > > print FILE "url_rewrite_program
>> > > > > > > /usr/sbin/redirect_wrapper\n";
>> > > > > > > - print FILE "url_rewrite_children
>> > > > > > > $proxysettings{'CHILDREN'}\n\n";
>> > > > > > > + print FILE "url_rewrite_children ",
>> > > > > > > &General::number_cpu_cores(), "\n\n";
>> > > > > > > }
>> > > > > > > 
>> > > > > > > # Include file with user defined settings.
> 
> 

Re: [PATCH] BUG 11786 - squid: Remove setting for filter processes the number of Squid processes

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 1732 bytes --]

On 30.10.2018 17:05, Michael Tremer wrote:
> On Tue, 2018-10-30 at 15:37 +0100, Matthias Fischer wrote:
>> On 30.10.2018 15:03, Michael Tremer wrote:
>> > Hey,

Hi,

> ...
> Looking at the options that we have (
> http://www.squid-cache.org/Doc/config/url_rewrite_children/), could you please
> test this:
> 
>   url_rewrite_children X startup=N idle=N queue-size=64
> 
> With N being the number of cores that your system has. I do not think that there
> is any point in trying to run more than N queries at a time.
> 
> But squid doesn't start anything when it starts up. startup= changes that and
> idle= makes sure that all processes remain active.
> 
> A queue size of 64 seems sensible to me because that is probably what the
> average tabloid newspaper page has. Maybe it should be 16 * N or so...
> ...
I tested this quite a while now. If I set 'queue-size' to 64, there will
be occasional warnings:

...
2018/12/15 14:10:59 kid1| comm_udp_sendto FD 8, (family=2) 127.0.0.1:53:
(1) Operation not permitted
2018/12/15 14:10:59 kid1| idnsSendQuery FD 8: sendto: (1) Operation not
permitted
2018/12/15 14:11:09 kid1| comm_udp_sendto FD 8, (family=2) 127.0.0.1:53:
(1) Operation not permitted
2018/12/15 14:11:09 kid1| idnsSendQuery FD 8: sendto: (1) Operation not
permitted
2018/12/15 14:11:19 kid1| comm_udp_sendto FD 8, (family=2) 127.0.0.1:53:
(1) Operation not permitted
2018/12/15 14:11:19 kid1| idnsSendQuery FD 8: sendto: (1) Operation not
permitted
...

With a value of '128' these warnings do not occur.

Running on Duo Box.

Corresponding 'squid.conf' line, latest 'squid 4.4', 32bit:

...
url_rewrite_children 2 startup=2 idle=2 queue-size=128
...

Best,
Matthias

Re: [PATCH] BUG 11786 - squid: Remove setting for filter processes the number of Squid processes

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 1171 bytes --]

Hi,

On 15.12.2018 14:24, Matthias Fischer wrote:
> On 30.10.2018 17:05, Michael Tremer wrote:
>> On Tue, 2018-10-30 at 15:37 +0100, Matthias Fischer wrote:
>>> On 30.10.2018 15:03, Michael Tremer wrote:
>>> > Hey,
> 
> Hi,
> 
>> ...

>> A queue size of 64 seems sensible to me because that is probably what the>> average tabloid newspaper page has. Maybe it should be 16 * N or so...

>> ...

> I tested this quite a while now. If I set 'queue-size' to 64, there will
> be occasional warnings:
> 
> ...
> 2018/12/15 14:10:59 kid1| comm_udp_sendto FD 8, (family=2) 127.0.0.1:53:
> (1) Operation not permitted
> 2018/12/15 14:10:59 kid1| idnsSendQuery FD 8: sendto: (1) Operation not
> permitted
> ...
> 
> With a value of '128' these warnings do not occur.
> 
> Running on Duo Box.
> 
> Corresponding 'squid.conf' line, latest 'squid 4.4', 32bit:
> 
> ...
> url_rewrite_children 2 startup=2 idle=2 queue-size=128
> ...

FYI: Apparently two specific DNS block rules which blocked direct DNS
requests were responsible for these error messages.

After deactivating these rules, squid runs fine with "queue-size=64"

Best,
Matthias