Re: [PATCH] BUG 11786 - squid: Remove setting for filter processes the number of Squid processes

[Matthias Fischer <matthias.fischer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 11160 bytes --]

Hi,

On 02.11.2018 18:07, Michael Tremer wrote:
> On Fri, 2018-11-02 at 14:13 +0000, Daniel Weismüller wrote:
>> Hi Matthias
>> 
>> I've tested your changes. But for me it didn't work.
>> 14:24:48	squid: 	Bungled /etc/squid/squid.conf line 136:
>> url_rewrite_children 8 startup=2 idle=2 queue-size=64
>> 
>> According to the squid pages the "queue-size" argument isn't availlable for
>> squid 3.5.
>> http://www.squid-cache.org/Versions/v3/3.5/cfgman/url_rewrite_children.html

Ups. Sorry, my fault! I didn't check this and my production machine runs
'squid 4.4'.

>> I changed the code to: print FILE "url_rewrite_children
>> ",&General::number_cpu_cores()," startup=",&General::number_cpu_cores(),"
>> idle=",&General::number_cpu_cores()," \n\n";
>> 
>> This seems to work with squid 3.5 and leads to this line in 'squid.conf':
>> url_rewrite_children 8 startup=8 idle=8

As far as I remember, this could me the bare minimum for 'squid 3.x'.

> It is quite important to increase the size of the queue, because in your case 16
> might be enough, but that isn't true if that code is executed on a system with
> only one core.
> 
> But I strongly agree with startup and idle being the number of cores too, so
> that always a constant number of redirectors is running.

FYI:
I've been using 'squid 4.4' now for over a year. Meanwhile this version
has been declared stable beside '3.5.28':

=> http://www.squid-cache.org/Versions/

Any thoughts of upgrading?

> -Michael
> 
>> 
>> As you can see, my Test-ipfire has 8 virtual cores, so I couldn't see the
>> above problems.

Any warnings in '/var/log/squid/cache.log'?

>> Thanks for testing it!

No problem -you're welcome! ;-)

Best,
Matthias

>> -
>> Daniel
>> 
>> 
>> Matthias Fischer – Tue, 30. October 2018 18:12
>> > On 30.10.2018 17:05, Michael Tremer wrote:
>> > > On Tue, 2018-10-30 at 15:37 +0100, Matthias Fischer wrote:
>> > > > On 30.10.2018 15:03, Michael Tremer wrote:
>> > > > > Hey,
>> > > > 
>> > > > Hello,
>> > > > 
>> > > > > do you have any idea why the redirector is busy?
>> > > > 
>> > > > Sorry: no. Long time ago, I searched for an optimal value but didn't
>> > > > find any useful links. Right now, I'm testing 'squid 4.4" and its always
>> > > > the same. Too few redirectors => warning.
>> > > > 
>> > > > I only found something like "If you see this warning, you should
>> > > > increase the number of child processes and restart (or reconfigure)
>> > > > Squid. If the queue size becomes twice the number of redirectors, Squid
>> > > > aborts with a fatal message."
>> > > 
>> > > The last bit is not very good. Why does squid not queue the requests long
>> > > enough? That means even CPU cores * 2 or 4 won't be enough. On a single-
>> > > core
>> > > machine that would only be 2 requests. LOL
>> > > 
>> > > Looking at the options that we have (
>> > > www.squid-cache.org/Doc/config/url_rewrite_children/), could you please
>> > > test this:
>> > > 
>> > > url_rewrite_children X startup=N idle=N queue-size=64
>> > > 
>> > > With N being the number of cores that your system has. I do not think that
>> > 
>> > there
>> > > is any point in trying to run more than N queries at a time.
>> > > 
>> > > But squid doesn't start anything when it starts up. startup= changes that
>> > 
>> > and
>> > > idle= makes sure that all processes remain active.
>> > 
>> > Yep. I had this already in my 'proxy.cgi'.
>> > 
>> > I applied the patches again and changed the 'url_rewrite_children'-line
>> > in 'proxy.cgi' to read:
>> > 
>> > print FILE "url_rewrite_children ", &General::number_cpu_cores(),"
>> > startup=2 idle=2 queue-size=64\n\n";
>> > 
>> > This leads to this line in 'squid.conf':
>> > url_rewrite_children 2 startup=2 idle=2 queue-size=64
>> > 
>> > No warnings so far. Testing ...
>> > 
>> > Best,
>> > Matthias
>> > 
>> > > A queue size of 64 seems sensible to me because that is probably what the
>> > > average tabloid newspaper page has. Maybe it should be 16 * N or so...
>> > > 
>> > > -Michael
>> > > 
>> > > > 
>> > > > Taken from:
>> > > > 
>> > 
>> > etutorials.org/Server+Administration/Squid.+The+definitive+guide/Chapter+11.
>> > +Redirectors/11.4+Configuring+Squid/
>> > > > 
>> > > > So my current "40"-value was found through "try and look if the warning
>> > > > comes again"...
>> > > > 
>> > > > Best,
>> > > > Matthias
>> > > > 
>> > > > 
>> > > > > -Michael
>> > > > > 
>> > > > > On Tue, 2018-10-30 at 14:54 +0100, Matthias Fischer wrote:
>> > > > > > Hi,
>> > > > > > 
>> > > > > > I tested this patch today and I fear the resulting value is too low.
>> > > > > > 
>> > > > > > For me, its leads to:
>> > > > > > 
>> > > > > > ***SNIP***
>> > > > > > ...
>> > > > > > 2018/10/30 14:42:43 kid1| WARNING: All 2/2 redirector processes are
>> > 
>> > busy.
>> > > > > > 2018/10/30 14:42:43 kid1| WARNING: 4 pending requests queued
>> > > > > > 2018/10/30 14:42:43 kid1| WARNING: Consider increasing the number of
>> > > > > > redirector processes in your config file.
>> > > > > > ...
>> > > > > > 
>> > > > > > Current setting is "40" (Duo box, 2 Cores, 'squidclamav, squidguard)
>> > > > > > 
>> > > > > > Best,
>> > > > > > Matthias
>> > > > > > 
>> > > > > > On 30.10.2018 12:06, Daniel Weismüller wrote:
>> > > > > > > I added a function to determine the number of cores.
>> > > > > > > Now the number of squid processes will be equal to the number of
>> > 
>> > logical
>> > > > > > > cores.
>> > > > > > > Further I removed the possibility of changing the number
>> > > > > > > of squid processes in the proxy.cgi
>> > > > > > > 
>> > > > > > > Signed-off-by: Daniel Weismüller <daniel.weismueller(a)ipfire.org>
>> > > > > > > ---
>> > > > > > > config/cfgroot/general-functions.pl | 7 +++++++
>> > > > > > > html/cgi-bin/proxy.cgi | 16 ++--------------
>> > > > > > > 2 files changed, 9 insertions(+), 14 deletions(-)
>> > > > > > > 
>> > > > > > > diff --git a/config/cfgroot/general-functions.pl
>> > > > > > > b/config/cfgroot/general-
>> > > > > > > functions.pl
>> > > > > > > index 0577afe28..e8495e885 100644
>> > > > > > > --- a/config/cfgroot/general-functions.pl
>> > > > > > > +++ b/config/cfgroot/general-functions.pl
>> > > > > > > @@ -1165,5 +1165,12 @@ sub dnssec_status() {
>> > > > > > > 
>> > > > > > > return $status;
>> > > > > > > }
>> > > > > > > +sub number_cpu_cores() {
>> > > > > > > + open my $cpuinfo, "/proc/cpuinfo" or die "Can't open cpuinfo:
>> > > > > > > $!\n";
>> > > > > > > + my $cores = scalar (map /^processor/, <$cpuinfo>);
>> > > > > > > + close $cpuinfo;
>> > > > > > > +
>> > > > > > > + return $cores;
>> > > > > > > +}
>> > > > > > > 
>> > > > > > > 1;
>> > > > > > > diff --git a/html/cgi-bin/proxy.cgi b/html/cgi-bin/proxy.cgi
>> > > > > > > index 738425b9a..92bebfe18 100644
>> > > > > > > --- a/html/cgi-bin/proxy.cgi
>> > > > > > > +++ b/html/cgi-bin/proxy.cgi
>> > > > > > > @@ -287,7 +287,6 @@ $proxysettings{'IDENT_USER_ACL'} = 'positive';
>> > > > > > > $proxysettings{'ENABLE_FILTER'} = 'off';
>> > > > > > > $proxysettings{'ENABLE_UPDXLRATOR'} = 'off';
>> > > > > > > $proxysettings{'ENABLE_CLAMAV'} = 'off';
>> > > > > > > -$proxysettings{'CHILDREN'} = '10';
>> > > > > > > 
>> > > > > > > $ncsa_buttontext = $Lang::tr{'advproxy NCSA create user'};
>> > > > > > > 
>> > > > > > > @@ -437,11 +436,6 @@ if (($proxysettings{'ACTION'} eq
>> > 
>> > $Lang::tr{'save'})
>> > > > > > > > > 
>> > > > > > > 
>> > > > > > > ($proxysettings{'ACTION'}
>> > > > > > > $errormessage = $Lang::tr{'invalid maximum incoming
>> > > > > > > size'};
>> > > > > > > goto ERROR;
>> > > > > > > }
>> > > > > > > - if (!($proxysettings{'CHILDREN'} =~ /^\d+$/) ||
>> > > > > > > ($proxysettings{'CHILDREN'} < 1))
>> > > > > > > - {
>> > > > > > > - $errormessage = $Lang::tr{'advproxy invalid num of
>> > > > > > > children'};
>> > > > > > > - goto ERROR;
>> > > > > > > - }
>> > > > > > > if ($proxysettings{'ENABLE_BROWSER_CHECK'} eq 'on')
>> > > > > > > {
>> > > > > > > $browser_regexp = '';
>> > > > > > > @@ -1034,12 +1028,8 @@ print <<END
>> > > > > > > </table>
>> > > > > > > <hr size='1'>
>> > > > > > > <table width='100%'>
>> > > > > > > -<tr><td class='base' colspan='4'><b>$Lang::tr{'advproxy
>> > > > > > > redirector
>> > > > > > > children'}</b></td></tr>
>> > > > > > > -<tr><td class='base' >$Lang::tr{'processes'}:&nbsp;<img
>> > 
>> > src='/blob.gif'
>> > > > > > > alt='*' /><input type='text' name='CHILDREN'
>> > > > > > > value='$proxysettings{'CHILDREN'}' size='5' /></td>
>> > > > > > > END
>> > > > > > > ;
>> > > > > > > -my $count = `ip n| wc -l`;
>> > > > > > > -if ( $count < 1 ){$count = 1;}
>> > > > > > > if ( -e "/usr/bin/squidclamav" ) {
>> > > > > > > print "<td class='base'><b>".$Lang::tr{'advproxy
>> > > > > > > squidclamav'}."</b><br
>> > > > > > > />";
>> > > > > > > if ( ! -e "/var/run/clamav/clamd.pid" ){
>> > > > > > > @@ -1048,18 +1038,16 @@ if ( -e "/usr/bin/squidclamav" ) {
>> > > > > > > }
>> > > > > > > else {
>> > > > > > > print $Lang::tr{'advproxy enabled'}."<input
>> > > > > > > type='checkbox'
>> > > > > > > name='ENABLE_CLAMAV' ".$checked{'ENABLE_CLAMAV'}{'on'}." /><br
>> > > > > > > />";
>> > > > > > > - print "+ ".int(( $count**(1/3)) * 8);}
>> > > > > > > +}
>> > > > > > > print "</td>";
>> > > > > > > } else {
>> > > > > > > print "<td></td>";
>> > > > > > > }
>> > > > > > > print "<td class='base'><a href='/cgi-
>> > > > > > > bin/urlfilter.cgi'><b>".$Lang::tr{'advproxy url
>> > > > > > > filter'}."</a></b><br
>> > > > > > > />";
>> > > > > > > print $Lang::tr{'advproxy enabled'}."<input type='checkbox'
>> > > > > > > name='ENABLE_FILTER' ".$checked{'ENABLE_FILTER'}{'on'}." /><br
>> > > > > > > />";
>> > > > > > > -print "+ ".int(($count**(1/3)) * 6);
>> > > > > > > print "</td>";
>> > > > > > > print "<td class='base'><a href='/cgi-
>> > > > > > > bin/updatexlrator.cgi'><b>".$Lang::tr{'advproxy update
>> > > > > > > accelerator'}."</a></b><br />";
>> > > > > > > print $Lang::tr{'advproxy enabled'}."<input type='checkbox'
>> > > > > > > name='ENABLE_UPDXLRATOR' ".$checked{'ENABLE_UPDXLRATOR'}{'on'}."
>> > 
>> > /><br
>> > > > > > > />";
>> > > > > > > -print "+ ".int(($count**(1/3)) * 5);
>> > > > > > > print "</td></tr>";
>> > > > > > > print <<END
>> > > > > > > </table>
>> > > > > > > @@ -4095,7 +4083,7 @@ END
>> > > > > > > if (($proxysettings{'ENABLE_FILTER'} eq 'on') ||
>> > > > > > > ($proxysettings{'ENABLE_UPDXLRATOR'} eq 'on') ||
>> > > > > > > ($proxysettings{'ENABLE_CLAMAV'} eq 'on'))
>> > > > > > > {
>> > > > > > > print FILE "url_rewrite_program
>> > > > > > > /usr/sbin/redirect_wrapper\n";
>> > > > > > > - print FILE "url_rewrite_children
>> > > > > > > $proxysettings{'CHILDREN'}\n\n";
>> > > > > > > + print FILE "url_rewrite_children ",
>> > > > > > > &General::number_cpu_cores(), "\n\n";
>> > > > > > > }
>> > > > > > > 
>> > > > > > > # Include file with user defined settings.
> 
>