Re: squid 5.1 gone stable

[Michael Tremer <michael.tremer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 5808 bytes --]

Hello,

> On 19 Aug 2021, at 17:32, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
> 
> Hi,
> 
> please comments below...
> 
> On 19.08.2021 15:57, Michael Tremer wrote:
>> Hello,
>> 
>>> On 18 Aug 2021, at 17:42, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
>>> 
>>> Hi,
>>> 
>>> On 16.08.2021 11:40, Michael Tremer wrote:
>>> ...
>>>>> What makes me wonder: during build, 'squid' says it can open '32768',
>>>>> during start its '4096'. If someone knows why, please enlighten me... ;-)
>>>> 4096 is the default maximum number of files any process can open at the same.
>>>> 
>>>> This is to protect the system from going crazy by having too many open files (because I think the file descriptor table used to be of a static size in older versions of the kernel).
>>>> 
>>>>>> I suppose this is enough and I can live with 32k. We should remove the field from the UI then.
>>>>> Me too, but are 4096 enough?
>>>> No. I don’t know why the squid team isn’t handling this better. We are hitting this problem every time we update to a new version.
>>>> 
>>>> I suppose this is fine for testing.
>>>> 
>>>> You can try adding “ulimit -n 32768” to the squid init script and then it should be able to open up to 32k files.
>>> ...
>>> 
>>> Thanks for the clarification - I tested this with 'squid 5.1'. It seems
>>> to work:
>>> 
>>> ...
>>> case "1$" in
>>> 	start)
>>> 	ulimit -n 32768
>>> 	getpids "squid"
>>> ...
>> 
>> What is “getpids” good for?
> 
> Its a function call from '/etc/init.d/functions', but please don't ask
> me what its good for: ;-)
> 
> ***SNIP***
> # This will ensure compatibility with previous LFS Bootscripts
> getpids()
> {
>   if [ -z "${PIDFILE}" ]; then
>      pidofproc -s -p "${PIDFILE}" $@
>   else
>     pidofproc -s $@
>   fi
>   base="${1##*/}"
> }
> ***SNAP***

It simply seems to print all PIDs of the squid processes.

>> Adding the ulimit call to the initscript and removing the configuration option from the CGI script is fine with me.
> 
> I'm testing. Works.
> 
> ***SNIP***
> 2021/08/19 18:14:58 kid1| Logfile: closing log
> stdio:/var/log/squid/access.log
> 2021/08/19 18:14:58 kid1| Open FD READ/WRITE 8 redirect_wrapper #1
> 2021/08/19 18:14:58 kid1| Open FD READ/WRITE 10 redirect_wrapper #2
> 2021/08/19 18:14:58 kid1| Squid Cache (Version 5.1): Exiting normally.
> 2021/08/19 18:14:58| Removing PID file (/var/run/squid.pid)
> 2021/08/19 18:15:08 kid1| Current Directory is /srv/web/ipfire/cgi-bin
> 2021/08/19 18:15:08 kid1| Creating missing swap directories
> 2021/08/19 18:15:08 kid1| No cache_dir stores are configured.
> 2021/08/19 18:15:08| Removing PID file (/var/run/squid.pid)
> 2021/08/19 18:15:09 kid1| Current Directory is /srv/web/ipfire/cgi-bin
> 2021/08/19 18:15:09 kid1| Starting Squid Cache version 5.1 for
> x86_64-pc-linux-gnu...
> 2021/08/19 18:15:09 kid1| Service Name: squid
> 2021/08/19 18:15:09 kid1| Process ID 27102
> 2021/08/19 18:15:09 kid1| Process Roles: worker
> 2021/08/19 18:15:09 kid1| With 32768 file descriptors available
> 2021/08/19 18:15:09 kid1| Initializing IP Cache...
> 2021/08/19 18:15:09 kid1| DNS Socket created at 0.0.0.0, FD 7
> 2021/08/19 18:15:09 kid1| Adding domain localdomain from /etc/resolv.conf
> 2021/08/19 18:15:09 kid1| Adding nameserver 127.0.0.1 from /etc/resolv.conf
> 2021/08/19 18:15:09 kid1| helperOpenServers: Starting 2/2
> 'redirect_wrapper' processes
> 2021/08/19 18:15:09 kid1| Logfile: opening log
> stdio:/var/log/squid/access.log
> 2021/08/19 18:15:09 kid1| Store logging disabled
> 2021/08/19 18:15:09 kid1| Swap maxSize 0 + 262144 KB, estimated 20164
> objects
> 2021/08/19 18:15:09 kid1| Target number of buckets: 1008
> 2021/08/19 18:15:09 kid1| Using 8192 Store buckets
> 2021/08/19 18:15:09 kid1| Max Mem size: 262144 KB
> 2021/08/19 18:15:09 kid1| Max Swap size: 0 KB
> 2021/08/19 18:15:09 kid1| Using Least Load store dir selection
> 2021/08/19 18:15:09 kid1| Current Directory is /srv/web/ipfire/cgi-bin
> 2021/08/19 18:15:09 kid1| Finished loading MIME types and icons.
> 2021/08/19 18:15:09 kid1| HTCP Disabled.
> 2021/08/19 18:15:09 kid1| Squid plugin modules loaded: 0
> 2021/08/19 18:15:09 kid1| Adaptation support is off.
> 2021/08/19 18:15:09 kid1| Accepting HTTP Socket connections at conn6
> local=192.168.100.254:8080 remote=[::] FD 13 flags=9
> 2021/08/19 18:15:09 kid1| Accepting HTTP Socket connections at conn8
> local=192.168.101.254:8080 remote=[::] FD 14 flags=9
> 2021/08/19 18:15:09 kid1| Accepting HTTP Socket connections at conn10
> local=127.0.0.1:8080 remote=[::] FD 15 flags=9
> 2021/08/19 18:15:10 kid1| storeLateRelease: released 0 objects
> ***SNAP***
> 
>>> For my 'cache_peer' problem I opened a bug report
>>> (https://bugs.squid-cache.org/show_bug.cgi?id=5147). Work in progress.
>> 
>> I didn’t get it, but I am sure you know what you are doing :)
> 
> Despite being "only" a "non-caching web proxy with advanced filtering
> capabilities for enhancing privacy", 'privoxy' still threw away up to
> 15-20% of unnecessary or unwanted ads or internet junk. I wanted to keep it.

Okay. Does that still work with all this HTTPS traffic?

>> It is good to work together with upstream.
> 
> Yep. And the 'squids' are friendly... ;-)
> 
> As I read the last answers (really fast reaction!), these messages about
> "failed" TCP connections are triggered by the rejected CONNECTs from
> 'privoxy'. "Squid v4 did not consider rejected CONNECTs a problem worth
> marking the peer DEAD for." They fixed this bug and now 'squid 5.1'
> does... And "the bad header field is sent by privoxy".
> 
> For now, I've disabled 'privoxy' to see how things are going without it.
> 
> Best,
> Matthias