Re: squid 5.1 gone stable

[Matthias Fischer <matthias.fischer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 5375 bytes --]

Hi,

please comments below...

On 19.08.2021 15:57, Michael Tremer wrote:
> Hello,
> 
>> On 18 Aug 2021, at 17:42, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
>> 
>> Hi,
>> 
>> On 16.08.2021 11:40, Michael Tremer wrote:
>> ...
>>>> What makes me wonder: during build, 'squid' says it can open '32768',
>>>> during start its '4096'. If someone knows why, please enlighten me... ;-)
>>> 4096 is the default maximum number of files any process can open at the same.
>>> 
>>> This is to protect the system from going crazy by having too many open files (because I think the file descriptor table used to be of a static size in older versions of the kernel).
>>> 
>>>>> I suppose this is enough and I can live with 32k. We should remove the field from the UI then.
>>>> Me too, but are 4096 enough?
>>> No. I don’t know why the squid team isn’t handling this better. We are hitting this problem every time we update to a new version.
>>> 
>>> I suppose this is fine for testing.
>>> 
>>> You can try adding “ulimit -n 32768” to the squid init script and then it should be able to open up to 32k files.
>> ...
>> 
>> Thanks for the clarification - I tested this with 'squid 5.1'. It seems
>> to work:
>> 
>> ...
>> case "1$" in
>> 	start)
>> 	ulimit -n 32768
>> 	getpids "squid"
>> ...
> 
> What is “getpids” good for?

Its a function call from '/etc/init.d/functions', but please don't ask
me what its good for: ;-)

***SNIP***
# This will ensure compatibility with previous LFS Bootscripts
getpids()
{
   if [ -z "${PIDFILE}" ]; then
      pidofproc -s -p "${PIDFILE}" $@
   else
     pidofproc -s $@
   fi
   base="${1##*/}"
}
***SNAP***

> Adding the ulimit call to the initscript and removing the configuration option from the CGI script is fine with me.

I'm testing. Works.

***SNIP***
2021/08/19 18:14:58 kid1| Logfile: closing log
stdio:/var/log/squid/access.log
2021/08/19 18:14:58 kid1| Open FD READ/WRITE 8 redirect_wrapper #1
2021/08/19 18:14:58 kid1| Open FD READ/WRITE 10 redirect_wrapper #2
2021/08/19 18:14:58 kid1| Squid Cache (Version 5.1): Exiting normally.
2021/08/19 18:14:58| Removing PID file (/var/run/squid.pid)
2021/08/19 18:15:08 kid1| Current Directory is /srv/web/ipfire/cgi-bin
2021/08/19 18:15:08 kid1| Creating missing swap directories
2021/08/19 18:15:08 kid1| No cache_dir stores are configured.
2021/08/19 18:15:08| Removing PID file (/var/run/squid.pid)
2021/08/19 18:15:09 kid1| Current Directory is /srv/web/ipfire/cgi-bin
2021/08/19 18:15:09 kid1| Starting Squid Cache version 5.1 for
x86_64-pc-linux-gnu...
2021/08/19 18:15:09 kid1| Service Name: squid
2021/08/19 18:15:09 kid1| Process ID 27102
2021/08/19 18:15:09 kid1| Process Roles: worker
2021/08/19 18:15:09 kid1| With 32768 file descriptors available
2021/08/19 18:15:09 kid1| Initializing IP Cache...
2021/08/19 18:15:09 kid1| DNS Socket created at 0.0.0.0, FD 7
2021/08/19 18:15:09 kid1| Adding domain localdomain from /etc/resolv.conf
2021/08/19 18:15:09 kid1| Adding nameserver 127.0.0.1 from /etc/resolv.conf
2021/08/19 18:15:09 kid1| helperOpenServers: Starting 2/2
'redirect_wrapper' processes
2021/08/19 18:15:09 kid1| Logfile: opening log
stdio:/var/log/squid/access.log
2021/08/19 18:15:09 kid1| Store logging disabled
2021/08/19 18:15:09 kid1| Swap maxSize 0 + 262144 KB, estimated 20164
objects
2021/08/19 18:15:09 kid1| Target number of buckets: 1008
2021/08/19 18:15:09 kid1| Using 8192 Store buckets
2021/08/19 18:15:09 kid1| Max Mem size: 262144 KB
2021/08/19 18:15:09 kid1| Max Swap size: 0 KB
2021/08/19 18:15:09 kid1| Using Least Load store dir selection
2021/08/19 18:15:09 kid1| Current Directory is /srv/web/ipfire/cgi-bin
2021/08/19 18:15:09 kid1| Finished loading MIME types and icons.
2021/08/19 18:15:09 kid1| HTCP Disabled.
2021/08/19 18:15:09 kid1| Squid plugin modules loaded: 0
2021/08/19 18:15:09 kid1| Adaptation support is off.
2021/08/19 18:15:09 kid1| Accepting HTTP Socket connections at conn6
local=192.168.100.254:8080 remote=[::] FD 13 flags=9
2021/08/19 18:15:09 kid1| Accepting HTTP Socket connections at conn8
local=192.168.101.254:8080 remote=[::] FD 14 flags=9
2021/08/19 18:15:09 kid1| Accepting HTTP Socket connections at conn10
local=127.0.0.1:8080 remote=[::] FD 15 flags=9
2021/08/19 18:15:10 kid1| storeLateRelease: released 0 objects
***SNAP***

>> For my 'cache_peer' problem I opened a bug report
>> (https://bugs.squid-cache.org/show_bug.cgi?id=5147). Work in progress.
> 
> I didn’t get it, but I am sure you know what you are doing :)

Despite being "only" a "non-caching web proxy with advanced filtering
capabilities for enhancing privacy", 'privoxy' still threw away up to
15-20% of unnecessary or unwanted ads or internet junk. I wanted to keep it.

> It is good to work together with upstream.

Yep. And the 'squids' are friendly... ;-)

As I read the last answers (really fast reaction!), these messages about
"failed" TCP connections are triggered by the rejected CONNECTs from
'privoxy'. "Squid v4 did not consider rejected CONNECTs a problem worth
marking the peer DEAD for." They fixed this bug and now 'squid 5.1'
does... And "the bad header field is sent by privoxy".

For now, I've disabled 'privoxy' to see how things are going without it.

Best,
Matthias