From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: [PATCH] BUG 11786 - squid: Remove setting for filter processes the number of Squid processes Date: Fri, 02 Nov 2018 17:07:29 +0000 Message-ID: In-Reply-To: <20181102141334.Horde.KF6fnxrU0E6eF96J1JWbZ90@whytea.ipfire-zuhause.de> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============4482176765023998832==" List-Id: --===============4482176765023998832== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable On Fri, 2018-11-02 at 14:13 +0000, Daniel Weism=C3=BCller wrote: > Hi Matthias >=20 > I've tested your changes. But for me it didn't work. > 14:24:48 squid: Bungled /etc/squid/squid.conf line 136: > url_rewrite_children 8 startup=3D2 idle=3D2 queue-size=3D64 >=20 > According to the squid pages the "queue-size" argument isn't availlable for > squid 3.5. > http://www.squid-cache.org/Versions/v3/3.5/cfgman/url_rewrite_children.html >=20 > I changed the code to: print FILE "url_rewrite_children > ",&General::number_cpu_cores()," startup=3D",&General::number_cpu_cores()," > idle=3D",&General::number_cpu_cores()," \n\n"; >=20 > This seems to work with squid 3.5 and leads to this line in 'squid.conf': > url_rewrite_children 8 startup=3D8 idle=3D8 It is quite important to increase the size of the queue, because in your case= 16 might be enough, but that isn't true if that code is executed on a system with only one core. But I strongly agree with startup and idle being the number of cores too, so that always a constant number of redirectors is running. -Michael >=20 > As you can see, my Test-ipfire has 8 virtual cores, so I couldn't see the > above problems. > Thanks for testing it! >=20 > - > Daniel >=20 >=20 > Matthias Fischer =E2=80=93 Tue, 30. October 2018 18:12 > > On 30.10.2018 17:05, Michael Tremer wrote: > > > On Tue, 2018-10-30 at 15:37 +0100, Matthias Fischer wrote: > > > > On 30.10.2018 15:03, Michael Tremer wrote: > > > > > Hey, > > > >=20 > > > > Hello, > > > >=20 > > > > > do you have any idea why the redirector is busy? > > > >=20 > > > > Sorry: no. Long time ago, I searched for an optimal value but didn't > > > > find any useful links. Right now, I'm testing 'squid 4.4" and its alw= ays > > > > the same. Too few redirectors =3D> warning. > > > >=20 > > > > I only found something like "If you see this warning, you should > > > > increase the number of child processes and restart (or reconfigure) > > > > Squid. If the queue size becomes twice the number of redirectors, Squ= id > > > > aborts with a fatal message." > > >=20 > > > The last bit is not very good. Why does squid not queue the requests lo= ng > > > enough? That means even CPU cores * 2 or 4 won't be enough. On a single- > > > core > > > machine that would only be 2 requests. LOL > > >=20 > > > Looking at the options that we have ( > > > www.squid-cache.org/Doc/config/url_rewrite_children/), could you please > > > test this: > > >=20 > > > url_rewrite_children X startup=3DN idle=3DN queue-size=3D64 > > >=20 > > > With N being the number of cores that your system has. I do not think t= hat > >=20 > > there > > > is any point in trying to run more than N queries at a time. > > >=20 > > > But squid doesn't start anything when it starts up. startup=3D changes = that > >=20 > > and > > > idle=3D makes sure that all processes remain active. > >=20 > > Yep. I had this already in my 'proxy.cgi'. > >=20 > > I applied the patches again and changed the 'url_rewrite_children'-line > > in 'proxy.cgi' to read: > >=20 > > print FILE "url_rewrite_children ", &General::number_cpu_cores()," > > startup=3D2 idle=3D2 queue-size=3D64\n\n"; > >=20 > > This leads to this line in 'squid.conf': > > url_rewrite_children 2 startup=3D2 idle=3D2 queue-size=3D64 > >=20 > > No warnings so far. Testing ... > >=20 > > Best, > > Matthias > >=20 > > > A queue size of 64 seems sensible to me because that is probably what t= he > > > average tabloid newspaper page has. Maybe it should be 16 * N or so... > > >=20 > > > -Michael > > >=20 > > > >=20 > > > > Taken from: > > > >=20 > >=20 > > etutorials.org/Server+Administration/Squid.+The+definitive+guide/Chapter+= 11. > > +Redirectors/11.4+Configuring+Squid/ > > > >=20 > > > > So my current "40"-value was found through "try and look if the warni= ng > > > > comes again"... > > > >=20 > > > > Best, > > > > Matthias > > > >=20 > > > >=20 > > > > > -Michael > > > > >=20 > > > > > On Tue, 2018-10-30 at 14:54 +0100, Matthias Fischer wrote: > > > > > > Hi, > > > > > >=20 > > > > > > I tested this patch today and I fear the resulting value is too l= ow. > > > > > >=20 > > > > > > For me, its leads to: > > > > > >=20 > > > > > > ***SNIP*** > > > > > > ... > > > > > > 2018/10/30 14:42:43 kid1| WARNING: All 2/2 redirector processes a= re > >=20 > > busy. > > > > > > 2018/10/30 14:42:43 kid1| WARNING: 4 pending requests queued > > > > > > 2018/10/30 14:42:43 kid1| WARNING: Consider increasing the number= of > > > > > > redirector processes in your config file. > > > > > > ... > > > > > >=20 > > > > > > Current setting is "40" (Duo box, 2 Cores, 'squidclamav, squidgua= rd) > > > > > >=20 > > > > > > Best, > > > > > > Matthias > > > > > >=20 > > > > > > On 30.10.2018 12:06, Daniel Weism=C3=BCller wrote: > > > > > > > I added a function to determine the number of cores. > > > > > > > Now the number of squid processes will be equal to the number of > >=20 > > logical > > > > > > > cores. > > > > > > > Further I removed the possibility of changing the number > > > > > > > of squid processes in the proxy.cgi > > > > > > >=20 > > > > > > > Signed-off-by: Daniel Weism=C3=BCller > > > > > > > --- > > > > > > > config/cfgroot/general-functions.pl | 7 +++++++ > > > > > > > html/cgi-bin/proxy.cgi | 16 ++-------------- > > > > > > > 2 files changed, 9 insertions(+), 14 deletions(-) > > > > > > >=20 > > > > > > > diff --git a/config/cfgroot/general-functions.pl > > > > > > > b/config/cfgroot/general- > > > > > > > functions.pl > > > > > > > index 0577afe28..e8495e885 100644 > > > > > > > --- a/config/cfgroot/general-functions.pl > > > > > > > +++ b/config/cfgroot/general-functions.pl > > > > > > > @@ -1165,5 +1165,12 @@ sub dnssec_status() { > > > > > > >=20 > > > > > > > return $status; > > > > > > > } > > > > > > > +sub number_cpu_cores() { > > > > > > > + open my $cpuinfo, "/proc/cpuinfo" or die "Can't open cpuinfo: > > > > > > > $!\n"; > > > > > > > + my $cores =3D scalar (map /^processor/, <$cpuinfo>); > > > > > > > + close $cpuinfo; > > > > > > > + > > > > > > > + return $cores; > > > > > > > +} > > > > > > >=20 > > > > > > > 1; > > > > > > > diff --git a/html/cgi-bin/proxy.cgi b/html/cgi-bin/proxy.cgi > > > > > > > index 738425b9a..92bebfe18 100644 > > > > > > > --- a/html/cgi-bin/proxy.cgi > > > > > > > +++ b/html/cgi-bin/proxy.cgi > > > > > > > @@ -287,7 +287,6 @@ $proxysettings{'IDENT_USER_ACL'} =3D 'posit= ive'; > > > > > > > $proxysettings{'ENABLE_FILTER'} =3D 'off'; > > > > > > > $proxysettings{'ENABLE_UPDXLRATOR'} =3D 'off'; > > > > > > > $proxysettings{'ENABLE_CLAMAV'} =3D 'off'; > > > > > > > -$proxysettings{'CHILDREN'} =3D '10'; > > > > > > >=20 > > > > > > > $ncsa_buttontext =3D $Lang::tr{'advproxy NCSA create user'}; > > > > > > >=20 > > > > > > > @@ -437,11 +436,6 @@ if (($proxysettings{'ACTION'} eq > >=20 > > $Lang::tr{'save'}) > > > > > > > > >=20 > > > > > > >=20 > > > > > > > ($proxysettings{'ACTION'} > > > > > > > $errormessage =3D $Lang::tr{'invalid maximum incoming > > > > > > > size'}; > > > > > > > goto ERROR; > > > > > > > } > > > > > > > - if (!($proxysettings{'CHILDREN'} =3D~ /^\d+$/) || > > > > > > > ($proxysettings{'CHILDREN'} < 1)) > > > > > > > - { > > > > > > > - $errormessage =3D $Lang::tr{'advproxy invalid num of > > > > > > > children'}; > > > > > > > - goto ERROR; > > > > > > > - } > > > > > > > if ($proxysettings{'ENABLE_BROWSER_CHECK'} eq 'on') > > > > > > > { > > > > > > > $browser_regexp =3D ''; > > > > > > > @@ -1034,12 +1028,8 @@ print < > > > > > > > > > > > > >
> > > > > > > > > > > > > > - > > > > > > > - > > > > > > > END > > > > > > > ; > > > > > > > -my $count =3D `ip n| wc -l`; > > > > > > > -if ( $count < 1 ){$count =3D 1;} > > > > > > > if ( -e "/usr/bin/squidclamav" ) { > > > > > > > print ""; > > > > > > > } else { > > > > > > > print ""; > > > > > > > } > > > > > > > print ""; > > > > > > > print ""; > > > > > > > print < > > > > > >
$Lang::tr{'advproxy > > > > > > > redirector > > > > > > > children'}
$Lang::tr{'processes'}:  >=20 > > src=3D'/blob.gif' > > > > > > > alt=3D'*' /> > > > > > > value=3D'$proxysettings{'CHILDREN'}' size=3D'5' />".$Lang::tr{'advproxy > > > > > > > squidclamav'}."
> > > > > > />"; > > > > > > > if ( ! -e "/var/run/clamav/clamd.pid" ){ > > > > > > > @@ -1048,18 +1038,16 @@ if ( -e "/usr/bin/squidclamav" ) { > > > > > > > } > > > > > > > else { > > > > > > > print $Lang::tr{'advproxy enabled'}." > > > > > > type=3D'checkbox' > > > > > > > name=3D'ENABLE_CLAMAV' ".$checked{'ENABLE_CLAMAV'}{'on'}." />
> > > > > > />"; > > > > > > > - print "+ ".int(( $count**(1/3)) * 8);} > > > > > > > +} > > > > > > > print "		 > > > > > > bin/urlfilter.cgi'>".$Lang::tr{'advproxy url > > > > > > > filter'}."
> > > > > > />"; > > > > > > > print $Lang::tr{'advproxy enabled'}." > > > > > > name=3D'ENABLE_FILTER' ".$checked{'ENABLE_FILTER'}{'on'}." />
> > > > > > />"; > > > > > > > -print "+ ".int(($count**(1/3)) * 6); > > > > > > > print "		 > > > > > > bin/updatexlrator.cgi'>".$Lang::tr{'advproxy update > > > > > > > accelerator'}."
"; > > > > > > > print $Lang::tr{'advproxy enabled'}." > > > > > > name=3D'ENABLE_UPDXLRATOR' ".$checked{'ENABLE_UPDXLRATOR'}{'on'= }." > >=20 > > />
> > > > > > />"; > > > > > > > -print "+ ".int(($count**(1/3)) * 5); > > > > > > > print "
> > > > > > > @@ -4095,7 +4083,7 @@ END > > > > > > > if (($proxysettings{'ENABLE_FILTER'} eq 'on') || > > > > > > > ($proxysettings{'ENABLE_UPDXLRATOR'} eq 'on') || > > > > > > > ($proxysettings{'ENABLE_CLAMAV'} eq 'on')) > > > > > > > { > > > > > > > print FILE "url_rewrite_program > > > > > > > /usr/sbin/redirect_wrapper\n"; > > > > > > > - print FILE "url_rewrite_children > > > > > > > $proxysettings{'CHILDREN'}\n\n"; > > > > > > > + print FILE "url_rewrite_children ", > > > > > > > &General::number_cpu_cores(), "\n\n"; > > > > > > > } > > > > > > >=20 > > > > > > > # Include file with user defined settings. --===============4482176765023998832==--