Re: squid 5.1 gone stable

[Matthias Fischer <matthias.fischer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 6589 bytes --]

Hi,

On 23.08.2021 15:29, Michael Tremer wrote:
> Hello,
> 
>> On 19 Aug 2021, at 17:32, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
>> 
>> Hi,
>> 
>> please comments below...
>> 
>> On 19.08.2021 15:57, Michael Tremer wrote:
>>> Hello,
>>> 
>>>> On 18 Aug 2021, at 17:42, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> On 16.08.2021 11:40, Michael Tremer wrote:
>>>> ...
>>>>>> What makes me wonder: during build, 'squid' says it can open '32768',
>>>>>> during start its '4096'. If someone knows why, please enlighten me... ;-)
>>>>> 4096 is the default maximum number of files any process can open at the same.
>>>>> 
>>>>> This is to protect the system from going crazy by having too many open files (because I think the file descriptor table used to be of a static size in older versions of the kernel).
>>>>> 
>>>>>>> I suppose this is enough and I can live with 32k. We should remove the field from the UI then.
>>>>>> Me too, but are 4096 enough?
>>>>> No. I don’t know why the squid team isn’t handling this better. We are hitting this problem every time we update to a new version.
>>>>> 
>>>>> I suppose this is fine for testing.
>>>>> 
>>>>> You can try adding “ulimit -n 32768” to the squid init script and then it should be able to open up to 32k files.
>>>> ...
>>>> 
>>>> Thanks for the clarification - I tested this with 'squid 5.1'. It seems
>>>> to work:
>>>> 
>>>> ...
>>>> case "1$" in
>>>> 	start)
>>>> 	ulimit -n 32768
>>>> 	getpids "squid"
>>>> ...
>>> 
>>> What is “getpids” good for?
>> 
>> Its a function call from '/etc/init.d/functions', but please don't ask
>> me what its good for: ;-)
>> 
>> ***SNIP***
>> # This will ensure compatibility with previous LFS Bootscripts
>> getpids()
>> {
>>   if [ -z "${PIDFILE}" ]; then
>>      pidofproc -s -p "${PIDFILE}" $@
>>   else
>>     pidofproc -s $@
>>   fi
>>   base="${1##*/}"
>> }
>> ***SNAP***
> 
> It simply seems to print all PIDs of the squid processes.

Thanks. I thought something like that but wasn't sure.

>>> Adding the ulimit call to the initscript and removing the configuration option from the CGI script is fine with me.
>> 
>> I'm testing. Works.
>> 
>> ***SNIP***
>> 2021/08/19 18:14:58 kid1| Logfile: closing log
>> stdio:/var/log/squid/access.log
>> 2021/08/19 18:14:58 kid1| Open FD READ/WRITE 8 redirect_wrapper #1
>> 2021/08/19 18:14:58 kid1| Open FD READ/WRITE 10 redirect_wrapper #2
>> 2021/08/19 18:14:58 kid1| Squid Cache (Version 5.1): Exiting normally.
>> 2021/08/19 18:14:58| Removing PID file (/var/run/squid.pid)
>> 2021/08/19 18:15:08 kid1| Current Directory is /srv/web/ipfire/cgi-bin
>> 2021/08/19 18:15:08 kid1| Creating missing swap directories
>> 2021/08/19 18:15:08 kid1| No cache_dir stores are configured.
>> 2021/08/19 18:15:08| Removing PID file (/var/run/squid.pid)
>> 2021/08/19 18:15:09 kid1| Current Directory is /srv/web/ipfire/cgi-bin
>> 2021/08/19 18:15:09 kid1| Starting Squid Cache version 5.1 for
>> x86_64-pc-linux-gnu...
>> 2021/08/19 18:15:09 kid1| Service Name: squid
>> 2021/08/19 18:15:09 kid1| Process ID 27102
>> 2021/08/19 18:15:09 kid1| Process Roles: worker
>> 2021/08/19 18:15:09 kid1| With 32768 file descriptors available
>> 2021/08/19 18:15:09 kid1| Initializing IP Cache...
>> 2021/08/19 18:15:09 kid1| DNS Socket created at 0.0.0.0, FD 7
>> 2021/08/19 18:15:09 kid1| Adding domain localdomain from /etc/resolv.conf
>> 2021/08/19 18:15:09 kid1| Adding nameserver 127.0.0.1 from /etc/resolv.conf
>> 2021/08/19 18:15:09 kid1| helperOpenServers: Starting 2/2
>> 'redirect_wrapper' processes
>> 2021/08/19 18:15:09 kid1| Logfile: opening log
>> stdio:/var/log/squid/access.log
>> 2021/08/19 18:15:09 kid1| Store logging disabled
>> 2021/08/19 18:15:09 kid1| Swap maxSize 0 + 262144 KB, estimated 20164
>> objects
>> 2021/08/19 18:15:09 kid1| Target number of buckets: 1008
>> 2021/08/19 18:15:09 kid1| Using 8192 Store buckets
>> 2021/08/19 18:15:09 kid1| Max Mem size: 262144 KB
>> 2021/08/19 18:15:09 kid1| Max Swap size: 0 KB
>> 2021/08/19 18:15:09 kid1| Using Least Load store dir selection
>> 2021/08/19 18:15:09 kid1| Current Directory is /srv/web/ipfire/cgi-bin
>> 2021/08/19 18:15:09 kid1| Finished loading MIME types and icons.
>> 2021/08/19 18:15:09 kid1| HTCP Disabled.
>> 2021/08/19 18:15:09 kid1| Squid plugin modules loaded: 0
>> 2021/08/19 18:15:09 kid1| Adaptation support is off.
>> 2021/08/19 18:15:09 kid1| Accepting HTTP Socket connections at conn6
>> local=192.168.100.254:8080 remote=[::] FD 13 flags=9
>> 2021/08/19 18:15:09 kid1| Accepting HTTP Socket connections at conn8
>> local=192.168.101.254:8080 remote=[::] FD 14 flags=9
>> 2021/08/19 18:15:09 kid1| Accepting HTTP Socket connections at conn10
>> local=127.0.0.1:8080 remote=[::] FD 15 flags=9
>> 2021/08/19 18:15:10 kid1| storeLateRelease: released 0 objects
>> ***SNAP***
>> 
>>>> For my 'cache_peer' problem I opened a bug report
>>>> (https://bugs.squid-cache.org/show_bug.cgi?id=5147). Work in progress.
>>> 
>>> I didn’t get it, but I am sure you know what you are doing :)
>> 
>> Despite being "only" a "non-caching web proxy with advanced filtering
>> capabilities for enhancing privacy", 'privoxy' still threw away up to
>> 15-20% of unnecessary or unwanted ads or internet junk. I wanted to keep it.
> 
> Okay. Does that still work with all this HTTPS traffic?

I had no problems with https. Never. Only the children sometimes
complained... ;-)
If you can trust the privoxy statistics, it is surprising how much
banners, ads, junk, webbugs, popups etc. are still filtered out. I added
some common blocklists (e.g, Fanboy, Malware, Easylist) for this. With
the last version I started testing it with 'brotli 1.0.9' support.
Worked for me. But as said, 'squid 5.1' has its problems with 'privoxy'.
I'm at it - right now in "privoxy-users(a)lists.privoxy.org".

Best,
Matthias

>>> It is good to work together with upstream.
>> 
>> Yep. And the 'squids' are friendly... ;-)
>> 
>> As I read the last answers (really fast reaction!), these messages about
>> "failed" TCP connections are triggered by the rejected CONNECTs from
>> 'privoxy'. "Squid v4 did not consider rejected CONNECTs a problem worth
>> marking the peer DEAD for." They fixed this bug and now 'squid 5.1'
>> does... And "the bad header field is sent by privoxy".
>> 
>> For now, I've disabled 'privoxy' to see how things are going without it.
>> 
>> Best,
>> Matthias
>