Re: squid 5.1 gone stable

[Matthias Fischer <matthias.fischer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 10382 bytes --]

Hi,

For the records...

I made a few start/restart tests with '4.16' and '5.1' today.

What is puzzling me => please pay attention to the filedescriptor messages:

'squid 4.16' - cache_log:
***SNIP***
2021/08/07 01:27:23 kid1| Squid Cache (Version 4.16): Exiting normally.
2021/08/07 01:27:23| Removing PID file (/var/run/squid.pid)
2021/08/07 01:27:23| Current Directory is /
2021/08/07 01:27:23| Current Directory is /
2021/08/07 01:27:35| Created PID file (/var/run/squid.pid)
2021/08/07 01:27:35 kid1| Current Directory is /
2021/08/07 01:27:35 kid1| Creating missing swap directories
2021/08/07 01:27:35 kid1| No cache_dir stores are configured.
2021/08/07 01:27:35| Removing PID file (/var/run/squid.pid)
2021/08/07 01:27:36| Created PID file (/var/run/squid.pid)
2021/08/07 01:27:36 kid1| Current Directory is /
2021/08/07 01:27:36 kid1| Starting Squid Cache version 4.16 for
x86_64-pc-linux-gnu...
2021/08/07 01:27:36 kid1| Service Name: squid
2021/08/07 01:27:36 kid1| Process ID 14593
2021/08/07 01:27:36 kid1| Process Roles: worker
*2021/08/07 01:27:36 kid1| With 16384 file descriptors available* <----!
2021/08/07 01:27:36 kid1| Initializing IP Cache...
2021/08/07 01:27:36 kid1| DNS Socket created at 0.0.0.0, FD 5
2021/08/07 01:27:36 kid1| Adding domain localdomain from /etc/resolv.conf
2021/08/07 01:27:36 kid1| Adding nameserver 127.0.0.1 from /etc/resolv.conf
2021/08/07 01:27:36 kid1| helperOpenServers: Starting 2/4
'redirect_wrapper' processes
2021/08/07 01:27:36 kid1| Logfile: opening log
stdio:/var/log/squid/access.log
2021/08/07 01:27:36 kid1| Store logging disabled
2021/08/07 01:27:36 kid1| Swap maxSize 0 + 262144 KB, estimated 20164
objects
2021/08/07 01:27:36 kid1| Target number of buckets: 1008
2021/08/07 01:27:36 kid1| Using 8192 Store buckets
2021/08/07 01:27:36 kid1| Max Mem size: 262144 KB
2021/08/07 01:27:36 kid1| Max Swap size: 0 KB
2021/08/07 01:27:36 kid1| Using Least Load store dir selection
2021/08/07 01:27:36 kid1| Current Directory is /
2021/08/07 01:27:36 kid1| Finished loading MIME types and icons.
2021/08/07 01:27:36 kid1| HTCP Disabled.
2021/08/07 01:27:36 kid1| Configuring Parent 127.0.0.1/8118/7
2021/08/07 01:27:36 kid1| Squid plugin modules loaded: 0
2021/08/07 01:27:36 kid1| Adaptation support is off.
2021/08/07 01:27:36 kid1| Accepting HTTP Socket connections at
local=192.168.100.254:8080 remote=[::] FD 13 flags=9
2021/08/07 01:27:36 kid1| Accepting HTTP Socket connections at
local=192.168.101.254:8080 remote=[::] FD 14 flags=9
2021/08/07 01:27:36 kid1| Accepting HTTP Socket connections at
local=127.0.0.1:8080 remote=[::] FD 15 flags=9
2021/08/07 01:27:36 kid1| Starting new redirector helpers...
2021/08/07 01:27:36 kid1| helperOpenServers: Starting 2/4
'redirect_wrapper' processes
2021/08/07 01:27:37 kid1| storeLateRelease: released 0 objects
***SNAP***

'squid 5.1 - cache_log - with *exactly* the same configuration:
***SNIP***
2021/08/07 09:25:29 kid1| Squid Cache (Version 4.16): Exiting normally.
2021/08/07 09:25:29| Removing PID file (/var/run/squid.pid)
2021/08/07 09:25:29| Current Directory is /
2021/08/07 09:25:29| Current Directory is /
2021/08/07 09:25:46 kid1| Current Directory is /
2021/08/07 09:25:46 kid1| Creating missing swap directories
2021/08/07 09:25:46 kid1| No cache_dir stores are configured.
2021/08/07 09:25:46| Removing PID file (/var/run/squid.pid)
2021/08/07 09:25:46 kid1| Current Directory is /
2021/08/07 09:25:46 kid1| Starting Squid Cache version 5.1 for
x86_64-pc-linux-gnu...
2021/08/07 09:25:46 kid1| Service Name: squid
2021/08/07 09:25:46 kid1| Process ID 8881
2021/08/07 09:25:46 kid1| Process Roles: worker
*2021/08/07 09:25:46 kid1| NOTICE: Could not increase the number of
filedescriptors*              <------------!!!!!
*2021/08/07 09:25:46 kid1| With 4096 file descriptors available*
2021/08/07 09:25:46 kid1| Initializing IP Cache...
2021/08/07 09:25:46 kid1| DNS Socket created at 0.0.0.0, FD 7
2021/08/07 09:25:46 kid1| Adding domain localdomain from /etc/resolv.conf
2021/08/07 09:25:46 kid1| Adding nameserver 127.0.0.1 from /etc/resolv.conf
2021/08/07 09:25:46 kid1| helperOpenServers: Starting 2/4
'redirect_wrapper' processes
2021/08/07 09:25:46 kid1| Logfile: opening log
stdio:/var/log/squid/access.log
2021/08/07 09:25:46 kid1| Store logging disabled
2021/08/07 09:25:46 kid1| Swap maxSize 0 + 262144 KB, estimated 20164
objects
2021/08/07 09:25:46 kid1| Target number of buckets: 1008
2021/08/07 09:25:46 kid1| Using 8192 Store buckets
2021/08/07 09:25:46 kid1| Max Mem size: 262144 KB
2021/08/07 09:25:46 kid1| Max Swap size: 0 KB
2021/08/07 09:25:46 kid1| Using Least Load store dir selection
2021/08/07 09:25:46 kid1| Current Directory is /
2021/08/07 09:25:46 kid1| Finished loading MIME types and icons.
2021/08/07 09:25:46 kid1| HTCP Disabled.
2021/08/07 09:25:46 kid1| Configuring Parent 127.0.0.1/8118/7
2021/08/07 09:25:46 kid1| Squid plugin modules loaded: 0
2021/08/07 09:25:46 kid1| Adaptation support is off.
2021/08/07 09:25:46 kid1| Accepting HTTP Socket connections at conn6
local=192.168.100.254:8080 remote=[::] FD 13 flags=9
2021/08/07 09:25:46 kid1| Accepting HTTP Socket connections at conn8
local=192.168.101.254:8080 remote=[::] FD 14 flags=9
2021/08/07 09:25:46 kid1| Accepting HTTP Socket connections at conn10
local=127.0.0.1:8080 remote=[::] FD 15 flags=9
2021/08/07 09:25:47 kid1| storeLateRelease: released 0 objects
*2021/08/07 09:27:33 kid1| TCP connection to 127.0.0.1/8118 failed
current master transaction: master53*  <----- thats 'privoxy'
2021/08/07 09:28:34 kid1| Starting new redirector helpers...
...

'privoxy' is another problem - besides, no errors can be detected during
browsing/operation. Everything seems to be ok. I just get these warnings
in 'cache_log'.

For 'privoxy' to work I had to add these lines in 'squid.conf', which -
until now - never made any problems with 4.x versions:

...
cache_peer 127.0.0.1 parent 8118 0 name=privoxy no-query no-digest
no-netdb-exchange default
never_direct allow all
...

If "someone" has any ideas what is causing this:
Hints are welcome... ;-)

Best,
Matthias


On 06.08.2021 19:55, Matthias Fischer wrote:
> Hi,
> 
> On 06.08.2021 12:46, Michael Tremer wrote:
>> Hi,
>> 
>>> On 5 Aug 2021, at 19:12, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
>>> 
>>> Hi,
>>> 
>>> On 02.08.2021 18:39, Michael Tremer wrote:
>>>> Hey,
>>>> 
>>>> That sounds good.
>>>> 
>>>> The change log lists a couple of changed and remove directives:
>>>> 
>>>>  http://www.squid-cache.org/Versions/v5/squid-5.1-RELEASENOTES.html#s3
>>>> 
>>>> Could you check if we are affected by any of this change?
>>> 
>>> Building was ok - installing was ok. I checked the release notes and on
>>> the first glance didn't find anything we would need to worry about. Will
>>> take a closer look on weekend.
>>> 
>>> But...:
>>> 
>>> 5.1 is permanently complaining about filedescriptors and I couldn't find
>>> the reason for this so far. Even if I reduce the number during build to
>>> 32768 (this is supposedly his maximum), it keeps complaining:
>>> 
>>> ...
>>> 2021/08/03 19:19:03 kid1| NOTICE: Could not increase the number of
>>> filedescriptors
>>> 2021/08/03 19:19:03 kid1| With 4096 file descriptors available
>>> ...
>> 
>> That must be the system’s limit then. If the process is being launched as root, it should be able to change its own limits.
> 
> Yes. I think so. Its weird. 'squid 4.16' - running with *exactly* the
> same configuration (under Core 158) - doesn't complain. Only '5.1'
> won't. I'm searching.
> 
>>> And - it seems that it doesn't like 'privoxy'. Everything seems to
>>> work(!), no seen problems(!), but with 'squid 5.1' I'm getting these
>>> sporadic messages in 'cache_log':
>> 
>> We don’t have privoxy. Is that a custom thing that you are running?
> 
> Yes.
> That one leftover from the old copfilter, ported to IFire. We once
> talked about this... I rewrote it as an addon for IPFire. Its running
> without any problems with the last 'squid 4.*' versions so far. But as
> above, 'squid 5.1' somehow doesn't like it anymore. 'privoxy' is still
> filtering and working, but I'm getting these complains from 'squid'. Hm.
> 
> Best,
> Matthias
> 
>>> ...
>>> 2021/08/03 19:27:17 kid1| helperOpenServers: Starting 2/4
>>> 'redirect_wrapper' processes
>>>    current master transaction: master88
>>> 2021/08/03 19:27:33 kid1| TCP connection to 127.0.0.1/8118 failed
>>>    current master transaction: master53
>>> 2021/08/03 19:27:34 kid1| TCP connection to 127.0.0.1/8118 failed
>>>    current master transaction: master53
>>> 2021/08/03 19:27:56 kid1| TCP connection to 127.0.0.1/8118 failed
>>>    current master transaction: master53
>>> ...
>>> 2021/08/03 19:59:34 kid1| TCP connection to 127.0.0.1/8118 failed
>>>    current master transaction: master53
>>> 2021/08/03 19:59:34 kid1| Detected DEAD Parent: privoxy
>>>    current master transaction: master53
>>> 2021/08/03 19:59:34 kid1| Detected REVIVED Parent: privoxy
>>>    current master transaction: master53
>>> 2021/08/03 19:59:34 kid1| TCP connection to 127.0.0.1/8118 failed
>>>    current master transaction: master53
>>> ...
>>> 
>>> Somehow I don't like this. Will report, if I find the reason.
>>> 
>>> Best,
>>> Matthias
>>> 
>>> 
>>>> -Michael
>>>> 
>>>>> On 2 Aug 2021, at 17:12, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> for the records...
>>>>> 
>>>>> 5.1 has gone stable:
>>>>> 
>>>>> => http://www.squid-cache.org/Versions/
>>>>> 
>>>>> Excerpt from changelog:
>>>>> "Changes in squid-5.1 (01 Aug 2021):
>>>>> 
>>>>> 	- Bug 4696: Fix leaky String move assignment operator
>>>>> 	- Fix ACL-related reconfiguration memory leak
>>>>> 	- Fix SSL-Bump reconfiguration leaking public key memory
>>>>> 	- Fix build on RISC-V
>>>>> 	- Fix build on Ubuntu 21.04
>>>>> 
>>>>> Changes in squid-5.0.7 (04 Jul 2021):
>>>>> 
>>>>> 	- Fix a helper logging issues
>>>>> 	- Fix some helper connection issues
>>>>> 	- Cleanup: remove much unused code
>>>>> 	- ... and all fixes from 4.16
>>>>> ..."
>>>>> 
>>>>> I'm at it.
>>>>> 
>>>>> Best,
>>>>> Matthias
>>>>> 
>>>> 
>>> 
>> 
>