From mboxrd@z Thu Jan 1 00:00:00 1970 From: Matthias Fischer To: development@lists.ipfire.org Subject: Re: [PATCH] BUG 11786 - squid: Remove setting for filter processes the number of Squid processes Date: Tue, 30 Oct 2018 18:12:19 +0100 Message-ID: <3e666588-af03-9ade-533d-6fb28ef26bb9@ipfire.org> In-Reply-To: <074ab32112cf6adc5728207f0823b9f5abd32173.camel@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============5939877443477675284==" List-Id: --===============5939877443477675284== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable On 30.10.2018 17:05, Michael Tremer wrote: > On Tue, 2018-10-30 at 15:37 +0100, Matthias Fischer wrote: >> On 30.10.2018 15:03, Michael Tremer wrote: >> > Hey, >>=20 >> Hello, >>=20 >> > do you have any idea why the redirector is busy? >>=20 >> Sorry: no. Long time ago, I searched for an optimal value but didn't >> find any useful links. Right now, I'm testing 'squid 4.4" and its always >> the same. Too few redirectors =3D> warning. >>=20 >> I only found something like "If you see this warning, you should >> increase the number of child processes and restart (or reconfigure) >> Squid. If the queue size becomes twice the number of redirectors, Squid >> aborts with a fatal message." >=20 > The last bit is not very good. Why does squid not queue the requests long > enough? That means even CPU cores * 2 or 4 won't be enough. On a single-core > machine that would only be 2 requests. LOL >=20 > Looking at the options that we have ( > http://www.squid-cache.org/Doc/config/url_rewrite_children/), could you ple= ase > test this: >=20 > url_rewrite_children X startup=3DN idle=3DN queue-size=3D64 >=20 > With N being the number of cores that your system has. I do not think that = there > is any point in trying to run more than N queries at a time. >=20 > But squid doesn't start anything when it starts up. startup=3D changes that= and > idle=3D makes sure that all processes remain active. Yep. I had this already in my 'proxy.cgi'. I applied the patches again and changed the 'url_rewrite_children'-line in 'proxy.cgi' to read: print FILE "url_rewrite_children ", &General::number_cpu_cores()," startup=3D2 idle=3D2 queue-size=3D64\n\n"; This leads to this line in 'squid.conf': url_rewrite_children 2 startup=3D2 idle=3D2 queue-size=3D64 No warnings so far. Testing ... Best, Matthias > A queue size of 64 seems sensible to me because that is probably what the > average tabloid newspaper page has. Maybe it should be 16 * N or so... >=20 > -Michael >=20 >>=20 >> Taken from: >>=20 > http://etutorials.org/Server+Administration/Squid.+The+definitive+guide/Cha= pter+11.+Redirectors/11.4+Configuring+Squid/ >>=20 >> So my current "40"-value was found through "try and look if the warning >> comes again"... >>=20 >> Best, >> Matthias >>=20 >>=20 >> > -Michael >> >=20 >> > On Tue, 2018-10-30 at 14:54 +0100, Matthias Fischer wrote: >> > > Hi, >> > >=20 >> > > I tested this patch today and I fear the resulting value is too low. >> > >=20 >> > > For me, its leads to: >> > >=20 >> > > ***SNIP*** >> > > ... >> > > 2018/10/30 14:42:43 kid1| WARNING: All 2/2 redirector processes are bu= sy. >> > > 2018/10/30 14:42:43 kid1| WARNING: 4 pending requests queued >> > > 2018/10/30 14:42:43 kid1| WARNING: Consider increasing the number of >> > > redirector processes in your config file. >> > > ... >> > >=20 >> > > Current setting is "40" (Duo box, 2 Cores, 'squidclamav, squidguard) >> > >=20 >> > > Best, >> > > Matthias >> > >=20 >> > > On 30.10.2018 12:06, Daniel Weism=C3=BCller wrote: >> > > > I added a function to determine the number of cores. >> > > > Now the number of squid processes will be equal to the number of log= ical >> > > > cores. >> > > > Further I removed the possibility of changing the number >> > > > of squid processes in the proxy.cgi >> > > >=20 >> > > > Signed-off-by: Daniel Weism=C3=BCller >> > > > --- >> > > > config/cfgroot/general-functions.pl | 7 +++++++ >> > > > html/cgi-bin/proxy.cgi | 16 ++-------------- >> > > > 2 files changed, 9 insertions(+), 14 deletions(-) >> > > >=20 >> > > > diff --git a/config/cfgroot/general-functions.pl >> > > > b/config/cfgroot/general- >> > > > functions.pl >> > > > index 0577afe28..e8495e885 100644 >> > > > --- a/config/cfgroot/general-functions.pl >> > > > +++ b/config/cfgroot/general-functions.pl >> > > > @@ -1165,5 +1165,12 @@ sub dnssec_status() { >> > > > =20 >> > > > return $status; >> > > > } >> > > > +sub number_cpu_cores() { >> > > > + open my $cpuinfo, "/proc/cpuinfo" or die "Can't open cpuinfo: >> > > > $!\n"; >> > > > + my $cores =3D scalar (map /^processor/, <$cpuinfo>); >> > > > + close $cpuinfo; >> > > > + >> > > > + return $cores; >> > > > +} >> > > > =20 >> > > > 1; >> > > > diff --git a/html/cgi-bin/proxy.cgi b/html/cgi-bin/proxy.cgi >> > > > index 738425b9a..92bebfe18 100644 >> > > > --- a/html/cgi-bin/proxy.cgi >> > > > +++ b/html/cgi-bin/proxy.cgi >> > > > @@ -287,7 +287,6 @@ $proxysettings{'IDENT_USER_ACL'} =3D 'positive'; >> > > > $proxysettings{'ENABLE_FILTER'} =3D 'off'; >> > > > $proxysettings{'ENABLE_UPDXLRATOR'} =3D 'off'; >> > > > $proxysettings{'ENABLE_CLAMAV'} =3D 'off'; >> > > > -$proxysettings{'CHILDREN'} =3D '10'; >> > > > =20 >> > > > $ncsa_buttontext =3D $Lang::tr{'advproxy NCSA create user'}; >> > > > =20 >> > > > @@ -437,11 +436,6 @@ if (($proxysettings{'ACTION'} eq $Lang::tr{'sav= e'}) >> > > > || >> > > > ($proxysettings{'ACTION'} >> > > > $errormessage =3D $Lang::tr{'invalid maximum incoming >> > > > size'}; >> > > > goto ERROR; >> > > > } >> > > > - if (!($proxysettings{'CHILDREN'} =3D~ /^\d+$/) || >> > > > ($proxysettings{'CHILDREN'} < 1)) >> > > > - { >> > > > - $errormessage =3D $Lang::tr{'advproxy invalid num of >> > > > children'}; >> > > > - goto ERROR; >> > > > - } >> > > > if ($proxysettings{'ENABLE_BROWSER_CHECK'} eq 'on') >> > > > { >> > > > $browser_regexp =3D ''; >> > > > @@ -1034,12 +1028,8 @@ print <> > > > >> > > >
>> > > > >> > > > - >> > > > - >> > > > END >> > > > ; >> > > > -my $count =3D `ip n| wc -l`; >> > > > -if ( $count < 1 ){$count =3D 1;} >> > > > if ( -e "/usr/bin/squidclamav" ) { >> > > > print ""; >> > > > } else { >> > > > print ""; >> > > > } >> > > > print ""; >> > > > print ""; >> > > > print <> > > >
$Lang::tr{'advproxy redirec= tor >> > > > children'}
$Lang::tr{'processes'}: > > > > alt=3D'*' />> > > > value=3D'$proxysettings{'CHILDREN'}' size=3D'5' />".$Lang::tr{'advproxy >> > > > squidclamav'}."
> > > > />"; >> > > > if ( ! -e "/var/run/clamav/clamd.pid" ){ >> > > > @@ -1048,18 +1038,16 @@ if ( -e "/usr/bin/squidclamav" ) { >> > > > } >> > > > else { >> > > > print $Lang::tr{'advproxy enabled'}."> > > > type=3D'checkbox' >> > > > name=3D'ENABLE_CLAMAV' ".$checked{'ENABLE_CLAMAV'}{'on'}." />
"; >> > > > - print "+ ".int(( $count**(1/3)) * 8);} >> > > > +} >> > > > print "		> > > > bin/urlfilter.cgi'>".$Lang::tr{'advproxy url filter'}."
> > > > />"; >> > > > print $Lang::tr{'advproxy enabled'}."> > > > name=3D'ENABLE_FILTER' ".$checked{'ENABLE_FILTER'}{'on'}." />
"; >> > > > -print "+ ".int(($count**(1/3)) * 6); >> > > > print "		> > > > bin/updatexlrator.cgi'>".$Lang::tr{'advproxy update >> > > > accelerator'}."
"; >> > > > print $Lang::tr{'advproxy enabled'}."> > > > name=3D'ENABLE_UPDXLRATOR' ".$checked{'ENABLE_UPDXLRATOR'}{'on'}." /= >
> > > > />"; >> > > > -print "+ ".int(($count**(1/3)) * 5); >> > > > print "
>> > > > @@ -4095,7 +4083,7 @@ END >> > > > if (($proxysettings{'ENABLE_FILTER'} eq 'on') || >> > > > ($proxysettings{'ENABLE_UPDXLRATOR'} eq 'on') || >> > > > ($proxysettings{'ENABLE_CLAMAV'} eq 'on')) >> > > > { >> > > > print FILE "url_rewrite_program >> > > > /usr/sbin/redirect_wrapper\n"; >> > > > - print FILE "url_rewrite_children >> > > > $proxysettings{'CHILDREN'}\n\n"; >> > > > + print FILE "url_rewrite_children ", >> > > > &General::number_cpu_cores(), "\n\n"; >> > > > } >> > > > =20 >> > > > # Include file with user defined settings. --===============5939877443477675284==--