From mboxrd@z Thu Jan 1 00:00:00 1970 From: Matthias Fischer To: development@lists.ipfire.org Subject: Re: squid 5.1 gone stable Date: Mon, 23 Aug 2021 18:25:28 +0200 Message-ID: In-Reply-To: <7A008831-7AAD-4A48-ABB9-3CC17F5C1CED@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============3840709841334542006==" List-Id: --===============3840709841334542006== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi, On 23.08.2021 15:29, Michael Tremer wrote: > Hello, >=20 >> On 19 Aug 2021, at 17:32, Matthias Fischer = wrote: >>=20 >> Hi, >>=20 >> please comments below... >>=20 >> On 19.08.2021 15:57, Michael Tremer wrote: >>> Hello, >>>=20 >>>> On 18 Aug 2021, at 17:42, Matthias Fischer wrote: >>>>=20 >>>> Hi, >>>>=20 >>>> On 16.08.2021 11:40, Michael Tremer wrote: >>>> ... >>>>>> What makes me wonder: during build, 'squid' says it can open '32768', >>>>>> during start its '4096'. If someone knows why, please enlighten me... = ;-) >>>>> 4096 is the default maximum number of files any process can open at the= same. >>>>>=20 >>>>> This is to protect the system from going crazy by having too many open = files (because I think the file descriptor table used to be of a static size = in older versions of the kernel). >>>>>=20 >>>>>>> I suppose this is enough and I can live with 32k. We should remove th= e field from the UI then. >>>>>> Me too, but are 4096 enough? >>>>> No. I don=E2=80=99t know why the squid team isn=E2=80=99t handling this= better. We are hitting this problem every time we update to a new version. >>>>>=20 >>>>> I suppose this is fine for testing. >>>>>=20 >>>>> You can try adding =E2=80=9Culimit -n 32768=E2=80=9D to the squid init = script and then it should be able to open up to 32k files. >>>> ... >>>>=20 >>>> Thanks for the clarification - I tested this with 'squid 5.1'. It seems >>>> to work: >>>>=20 >>>> ... >>>> case "1$" in >>>> start) >>>> ulimit -n 32768 >>>> getpids "squid" >>>> ... >>>=20 >>> What is =E2=80=9Cgetpids=E2=80=9D good for? >>=20 >> Its a function call from '/etc/init.d/functions', but please don't ask >> me what its good for: ;-) >>=20 >> ***SNIP*** >> # This will ensure compatibility with previous LFS Bootscripts >> getpids() >> { >> if [ -z "${PIDFILE}" ]; then >> pidofproc -s -p "${PIDFILE}" $@ >> else >> pidofproc -s $@ >> fi >> base=3D"${1##*/}" >> } >> ***SNAP*** >=20 > It simply seems to print all PIDs of the squid processes. Thanks. I thought something like that but wasn't sure. >>> Adding the ulimit call to the initscript and removing the configuration o= ption from the CGI script is fine with me. >>=20 >> I'm testing. Works. >>=20 >> ***SNIP*** >> 2021/08/19 18:14:58 kid1| Logfile: closing log >> stdio:/var/log/squid/access.log >> 2021/08/19 18:14:58 kid1| Open FD READ/WRITE 8 redirect_wrapper #1 >> 2021/08/19 18:14:58 kid1| Open FD READ/WRITE 10 redirect_wrapper #2 >> 2021/08/19 18:14:58 kid1| Squid Cache (Version 5.1): Exiting normally. >> 2021/08/19 18:14:58| Removing PID file (/var/run/squid.pid) >> 2021/08/19 18:15:08 kid1| Current Directory is /srv/web/ipfire/cgi-bin >> 2021/08/19 18:15:08 kid1| Creating missing swap directories >> 2021/08/19 18:15:08 kid1| No cache_dir stores are configured. >> 2021/08/19 18:15:08| Removing PID file (/var/run/squid.pid) >> 2021/08/19 18:15:09 kid1| Current Directory is /srv/web/ipfire/cgi-bin >> 2021/08/19 18:15:09 kid1| Starting Squid Cache version 5.1 for >> x86_64-pc-linux-gnu... >> 2021/08/19 18:15:09 kid1| Service Name: squid >> 2021/08/19 18:15:09 kid1| Process ID 27102 >> 2021/08/19 18:15:09 kid1| Process Roles: worker >> 2021/08/19 18:15:09 kid1| With 32768 file descriptors available >> 2021/08/19 18:15:09 kid1| Initializing IP Cache... >> 2021/08/19 18:15:09 kid1| DNS Socket created at 0.0.0.0, FD 7 >> 2021/08/19 18:15:09 kid1| Adding domain localdomain from /etc/resolv.conf >> 2021/08/19 18:15:09 kid1| Adding nameserver 127.0.0.1 from /etc/resolv.conf >> 2021/08/19 18:15:09 kid1| helperOpenServers: Starting 2/2 >> 'redirect_wrapper' processes >> 2021/08/19 18:15:09 kid1| Logfile: opening log >> stdio:/var/log/squid/access.log >> 2021/08/19 18:15:09 kid1| Store logging disabled >> 2021/08/19 18:15:09 kid1| Swap maxSize 0 + 262144 KB, estimated 20164 >> objects >> 2021/08/19 18:15:09 kid1| Target number of buckets: 1008 >> 2021/08/19 18:15:09 kid1| Using 8192 Store buckets >> 2021/08/19 18:15:09 kid1| Max Mem size: 262144 KB >> 2021/08/19 18:15:09 kid1| Max Swap size: 0 KB >> 2021/08/19 18:15:09 kid1| Using Least Load store dir selection >> 2021/08/19 18:15:09 kid1| Current Directory is /srv/web/ipfire/cgi-bin >> 2021/08/19 18:15:09 kid1| Finished loading MIME types and icons. >> 2021/08/19 18:15:09 kid1| HTCP Disabled. >> 2021/08/19 18:15:09 kid1| Squid plugin modules loaded: 0 >> 2021/08/19 18:15:09 kid1| Adaptation support is off. >> 2021/08/19 18:15:09 kid1| Accepting HTTP Socket connections at conn6 >> local=3D192.168.100.254:8080 remote=3D[::] FD 13 flags=3D9 >> 2021/08/19 18:15:09 kid1| Accepting HTTP Socket connections at conn8 >> local=3D192.168.101.254:8080 remote=3D[::] FD 14 flags=3D9 >> 2021/08/19 18:15:09 kid1| Accepting HTTP Socket connections at conn10 >> local=3D127.0.0.1:8080 remote=3D[::] FD 15 flags=3D9 >> 2021/08/19 18:15:10 kid1| storeLateRelease: released 0 objects >> ***SNAP*** >>=20 >>>> For my 'cache_peer' problem I opened a bug report >>>> (https://bugs.squid-cache.org/show_bug.cgi?id=3D5147). Work in progress. >>>=20 >>> I didn=E2=80=99t get it, but I am sure you know what you are doing :) >>=20 >> Despite being "only" a "non-caching web proxy with advanced filtering >> capabilities for enhancing privacy", 'privoxy' still threw away up to >> 15-20% of unnecessary or unwanted ads or internet junk. I wanted to keep i= t. >=20 > Okay. Does that still work with all this HTTPS traffic? I had no problems with https. Never. Only the children sometimes complained... ;-) If you can trust the privoxy statistics, it is surprising how much banners, ads, junk, webbugs, popups etc. are still filtered out. I added some common blocklists (e.g, Fanboy, Malware, Easylist) for this. With the last version I started testing it with 'brotli 1.0.9' support. Worked for me. But as said, 'squid 5.1' has its problems with 'privoxy'. I'm at it - right now in "privoxy-users(a)lists.privoxy.org". Best, Matthias >>> It is good to work together with upstream. >>=20 >> Yep. And the 'squids' are friendly... ;-) >>=20 >> As I read the last answers (really fast reaction!), these messages about >> "failed" TCP connections are triggered by the rejected CONNECTs from >> 'privoxy'. "Squid v4 did not consider rejected CONNECTs a problem worth >> marking the peer DEAD for." They fixed this bug and now 'squid 5.1' >> does... And "the bad header field is sent by privoxy". >>=20 >> For now, I've disabled 'privoxy' to see how things are going without it. >>=20 >> Best, >> Matthias >=20 --===============3840709841334542006==--