squid 5.1 gone stable

squid 5.1 gone stable

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 1194 bytes --]

Hi,

...for the records...: ;-)

Today I tested building 'squid 5.1' with our usual configure options:

...
--with-dl \
--with-filedescriptors=$(( 16384 * 64 )) \
--with-large-files \
--without-gnutls \
...

No errors in '_build.ipfire.log' for this 'squid' - except:

...
checking for library containing log... none required
configure: forcing default of 1048576 filedescriptors (user-forced)
checking Default FD_SETSIZE value... 1024
checking for getrlimit... yes
checking for setrlimit... yes
checking Maximum number of filedescriptors we can open... 32768
configure: Default number of filedescriptors: 1048576
...

So the maximum number of filedescriptors which are possible for 'squid
5.1' are 32768!?

Ok then. I rebuilt the whole thing with the "maximum":

...
--with-filedescriptors=32768 \
...

But no change. During starting, 'squid 5.1.' still complains:
...
NOTICE: Could not increase the number of filedescriptors
With 4096 file descriptors available
...

This is the value reported by 'ulimit -n' on my IPFire / Core 158.

Currently, only 'squid 4.16' can increase this number under *exactly*
the same environment.

What consequences could it have, respectively!?

Best,
Matthias

Re: squid 5.1 gone stable

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1614 bytes --]

Hello,

> On 7 Aug 2021, at 17:13, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
> 
> Hi,
> 
> ...for the records...: ;-)
> 
> Today I tested building 'squid 5.1' with our usual configure options:
> 
> ...
> --with-dl \
> --with-filedescriptors=$(( 16384 * 64 )) \
> --with-large-files \
> --without-gnutls \
> ...
> 
> No errors in '_build.ipfire.log' for this 'squid' - except:
> 
> ...
> checking for library containing log... none required
> configure: forcing default of 1048576 filedescriptors (user-forced)
> checking Default FD_SETSIZE value... 1024
> checking for getrlimit... yes
> checking for setrlimit... yes
> checking Maximum number of filedescriptors we can open... 32768
> configure: Default number of filedescriptors: 1048576
> ...
> 
> So the maximum number of filedescriptors which are possible for 'squid
> 5.1' are 32768!?
> 
> Ok then. I rebuilt the whole thing with the "maximum":
> 
> ...
> --with-filedescriptors=32768 \
> ...
> 
> But no change. During starting, 'squid 5.1.' still complains:
> ...
> NOTICE: Could not increase the number of filedescriptors
> With 4096 file descriptors available
> ...
> 
> This is the value reported by 'ulimit -n' on my IPFire / Core 158.
> 
> Currently, only 'squid 4.16' can increase this number under *exactly*
> the same environment.
> 
> What consequences could it have, respectively!?

This is the maximum amount of connections squid can open.

I suppose this is enough and I can live with 32k. We should remove the field from the UI then.

-Michael

> 
> Best,
> Matthias

Re: squid 5.1 gone stable

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 1970 bytes --]

Hi,

On 13.08.2021 11:22, Michael Tremer wrote:
> Hello,
> 
>> On 7 Aug 2021, at 17:13, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
>> 
>> Hi,
>> 
>> ...for the records...: ;-)
>> 
>> Today I tested building 'squid 5.1' with our usual configure options:
>> 
>> ...
>> --with-dl \
>> --with-filedescriptors=$(( 16384 * 64 )) \
>> --with-large-files \
>> --without-gnutls \
>> ...
>> 
>> No errors in '_build.ipfire.log' for this 'squid' - except:
>> 
>> ...
>> checking for library containing log... none required
>> configure: forcing default of 1048576 filedescriptors (user-forced)
>> checking Default FD_SETSIZE value... 1024
>> checking for getrlimit... yes
>> checking for setrlimit... yes
>> checking Maximum number of filedescriptors we can open... 32768
>> configure: Default number of filedescriptors: 1048576
>> ...
>> 
>> So the maximum number of filedescriptors which are possible for 'squid
>> 5.1' are 32768!?
>> 
>> Ok then. I rebuilt the whole thing with the "maximum":
>> 
>> ...
>> --with-filedescriptors=32768 \
>> ...
>> 
>> But no change. During starting, 'squid 5.1.' still complains:
>> ...
>> NOTICE: Could not increase the number of filedescriptors
>> With 4096 file descriptors available
>> ...
>> 
>> This is the value reported by 'ulimit -n' on my IPFire / Core 158.
>> 
>> Currently, only 'squid 4.16' can increase this number under *exactly*
>> the same environment.
>> 
>> What consequences could it have, respectively!?
> 
> This is the maximum amount of connections squid can open.

What makes me wonder: during build, 'squid' says it can open '32768',
during start its '4096'. If someone knows why, please enlighten me... ;-)

> I suppose this is enough and I can live with 32k. We should remove the field from the UI then.

Me too, but are 4096 enough?

Besides, I would wait to push '5.1' - I even didn't see an official
annoncement yet.

Best,
Matthias

Re: squid 5.1 gone stable

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 2726 bytes --]

Hello,

> On 15 Aug 2021, at 08:53, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
> 
> Hi,
> 
> On 13.08.2021 11:22, Michael Tremer wrote:
>> Hello,
>> 
>>> On 7 Aug 2021, at 17:13, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
>>> 
>>> Hi,
>>> 
>>> ...for the records...: ;-)
>>> 
>>> Today I tested building 'squid 5.1' with our usual configure options:
>>> 
>>> ...
>>> --with-dl \
>>> --with-filedescriptors=$(( 16384 * 64 )) \
>>> --with-large-files \
>>> --without-gnutls \
>>> ...
>>> 
>>> No errors in '_build.ipfire.log' for this 'squid' - except:
>>> 
>>> ...
>>> checking for library containing log... none required
>>> configure: forcing default of 1048576 filedescriptors (user-forced)
>>> checking Default FD_SETSIZE value... 1024
>>> checking for getrlimit... yes
>>> checking for setrlimit... yes
>>> checking Maximum number of filedescriptors we can open... 32768
>>> configure: Default number of filedescriptors: 1048576
>>> ...
>>> 
>>> So the maximum number of filedescriptors which are possible for 'squid
>>> 5.1' are 32768!?
>>> 
>>> Ok then. I rebuilt the whole thing with the "maximum":
>>> 
>>> ...
>>> --with-filedescriptors=32768 \
>>> ...
>>> 
>>> But no change. During starting, 'squid 5.1.' still complains:
>>> ...
>>> NOTICE: Could not increase the number of filedescriptors
>>> With 4096 file descriptors available
>>> ...
>>> 
>>> This is the value reported by 'ulimit -n' on my IPFire / Core 158.
>>> 
>>> Currently, only 'squid 4.16' can increase this number under *exactly*
>>> the same environment.
>>> 
>>> What consequences could it have, respectively!?
>> 
>> This is the maximum amount of connections squid can open.
> 
> What makes me wonder: during build, 'squid' says it can open '32768',
> during start its '4096'. If someone knows why, please enlighten me... ;-)

4096 is the default maximum number of files any process can open at the same.

This is to protect the system from going crazy by having too many open files (because I think the file descriptor table used to be of a static size in older versions of the kernel).

>> I suppose this is enough and I can live with 32k. We should remove the field from the UI then.
> 
> Me too, but are 4096 enough?

No. I don’t know why the squid team isn’t handling this better. We are hitting this problem every time we update to a new version.

I suppose this is fine for testing.

You can try adding “ulimit -n 32768” to the squid init script and then it should be able to open up to 32k files.

-Michael

> Besides, I would wait to push '5.1' - I even didn't see an official
> annoncement yet.
> 
> Best,
> Matthias

Re: squid 5.1 gone stable

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 1226 bytes --]

Hi,

On 16.08.2021 11:40, Michael Tremer wrote:
...
>> What makes me wonder: during build, 'squid' says it can open '32768',
>> during start its '4096'. If someone knows why, please enlighten me... ;-)
> 4096 is the default maximum number of files any process can open at the same.
> 
> This is to protect the system from going crazy by having too many open files (because I think the file descriptor table used to be of a static size in older versions of the kernel).
> 
>>> I suppose this is enough and I can live with 32k. We should remove the field from the UI then.
>> Me too, but are 4096 enough?
> No. I don’t know why the squid team isn’t handling this better. We are hitting this problem every time we update to a new version.
> 
> I suppose this is fine for testing.
> 
> You can try adding “ulimit -n 32768” to the squid init script and then it should be able to open up to 32k files.
...

Thanks for the clarification - I tested this with 'squid 5.1'. It seems
to work:

...
case "1$" in
	start)
	ulimit -n 32768
	getpids "squid"
...

For my 'cache_peer' problem I opened a bug report
(https://bugs.squid-cache.org/show_bug.cgi?id=5147). Work in progress.

Best,
Matthias

Re: squid 5.1 gone stable

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1661 bytes --]

Hello,

> On 18 Aug 2021, at 17:42, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
> 
> Hi,
> 
> On 16.08.2021 11:40, Michael Tremer wrote:
> ...
>>> What makes me wonder: during build, 'squid' says it can open '32768',
>>> during start its '4096'. If someone knows why, please enlighten me... ;-)
>> 4096 is the default maximum number of files any process can open at the same.
>> 
>> This is to protect the system from going crazy by having too many open files (because I think the file descriptor table used to be of a static size in older versions of the kernel).
>> 
>>>> I suppose this is enough and I can live with 32k. We should remove the field from the UI then.
>>> Me too, but are 4096 enough?
>> No. I don’t know why the squid team isn’t handling this better. We are hitting this problem every time we update to a new version.
>> 
>> I suppose this is fine for testing.
>> 
>> You can try adding “ulimit -n 32768” to the squid init script and then it should be able to open up to 32k files.
> ...
> 
> Thanks for the clarification - I tested this with 'squid 5.1'. It seems
> to work:
> 
> ...
> case "1$" in
> 	start)
> 	ulimit -n 32768
> 	getpids "squid"
> ...

What is “getpids” good for?

Adding the ulimit call to the initscript and removing the configuration option from the CGI script is fine with me.

> For my 'cache_peer' problem I opened a bug report
> (https://bugs.squid-cache.org/show_bug.cgi?id=5147). Work in progress.

I didn’t get it, but I am sure you know what you are doing :)

It is good to work together with upstream.

-Michael

> 
> Best,
> Matthias

Re: squid 5.1 gone stable

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 5375 bytes --]

Hi,

please comments below...

On 19.08.2021 15:57, Michael Tremer wrote:
> Hello,
> 
>> On 18 Aug 2021, at 17:42, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
>> 
>> Hi,
>> 
>> On 16.08.2021 11:40, Michael Tremer wrote:
>> ...
>>>> What makes me wonder: during build, 'squid' says it can open '32768',
>>>> during start its '4096'. If someone knows why, please enlighten me... ;-)
>>> 4096 is the default maximum number of files any process can open at the same.
>>> 
>>> This is to protect the system from going crazy by having too many open files (because I think the file descriptor table used to be of a static size in older versions of the kernel).
>>> 
>>>>> I suppose this is enough and I can live with 32k. We should remove the field from the UI then.
>>>> Me too, but are 4096 enough?
>>> No. I don’t know why the squid team isn’t handling this better. We are hitting this problem every time we update to a new version.
>>> 
>>> I suppose this is fine for testing.
>>> 
>>> You can try adding “ulimit -n 32768” to the squid init script and then it should be able to open up to 32k files.
>> ...
>> 
>> Thanks for the clarification - I tested this with 'squid 5.1'. It seems
>> to work:
>> 
>> ...
>> case "1$" in
>> 	start)
>> 	ulimit -n 32768
>> 	getpids "squid"
>> ...
> 
> What is “getpids” good for?

Its a function call from '/etc/init.d/functions', but please don't ask
me what its good for: ;-)

***SNIP***
# This will ensure compatibility with previous LFS Bootscripts
getpids()
{
   if [ -z "${PIDFILE}" ]; then
      pidofproc -s -p "${PIDFILE}" $@
   else
     pidofproc -s $@
   fi
   base="${1##*/}"
}
***SNAP***

> Adding the ulimit call to the initscript and removing the configuration option from the CGI script is fine with me.

I'm testing. Works.

***SNIP***
2021/08/19 18:14:58 kid1| Logfile: closing log
stdio:/var/log/squid/access.log
2021/08/19 18:14:58 kid1| Open FD READ/WRITE 8 redirect_wrapper #1
2021/08/19 18:14:58 kid1| Open FD READ/WRITE 10 redirect_wrapper #2
2021/08/19 18:14:58 kid1| Squid Cache (Version 5.1): Exiting normally.
2021/08/19 18:14:58| Removing PID file (/var/run/squid.pid)
2021/08/19 18:15:08 kid1| Current Directory is /srv/web/ipfire/cgi-bin
2021/08/19 18:15:08 kid1| Creating missing swap directories
2021/08/19 18:15:08 kid1| No cache_dir stores are configured.
2021/08/19 18:15:08| Removing PID file (/var/run/squid.pid)
2021/08/19 18:15:09 kid1| Current Directory is /srv/web/ipfire/cgi-bin
2021/08/19 18:15:09 kid1| Starting Squid Cache version 5.1 for
x86_64-pc-linux-gnu...
2021/08/19 18:15:09 kid1| Service Name: squid
2021/08/19 18:15:09 kid1| Process ID 27102
2021/08/19 18:15:09 kid1| Process Roles: worker
2021/08/19 18:15:09 kid1| With 32768 file descriptors available
2021/08/19 18:15:09 kid1| Initializing IP Cache...
2021/08/19 18:15:09 kid1| DNS Socket created at 0.0.0.0, FD 7
2021/08/19 18:15:09 kid1| Adding domain localdomain from /etc/resolv.conf
2021/08/19 18:15:09 kid1| Adding nameserver 127.0.0.1 from /etc/resolv.conf
2021/08/19 18:15:09 kid1| helperOpenServers: Starting 2/2
'redirect_wrapper' processes
2021/08/19 18:15:09 kid1| Logfile: opening log
stdio:/var/log/squid/access.log
2021/08/19 18:15:09 kid1| Store logging disabled
2021/08/19 18:15:09 kid1| Swap maxSize 0 + 262144 KB, estimated 20164
objects
2021/08/19 18:15:09 kid1| Target number of buckets: 1008
2021/08/19 18:15:09 kid1| Using 8192 Store buckets
2021/08/19 18:15:09 kid1| Max Mem size: 262144 KB
2021/08/19 18:15:09 kid1| Max Swap size: 0 KB
2021/08/19 18:15:09 kid1| Using Least Load store dir selection
2021/08/19 18:15:09 kid1| Current Directory is /srv/web/ipfire/cgi-bin
2021/08/19 18:15:09 kid1| Finished loading MIME types and icons.
2021/08/19 18:15:09 kid1| HTCP Disabled.
2021/08/19 18:15:09 kid1| Squid plugin modules loaded: 0
2021/08/19 18:15:09 kid1| Adaptation support is off.
2021/08/19 18:15:09 kid1| Accepting HTTP Socket connections at conn6
local=192.168.100.254:8080 remote=[::] FD 13 flags=9
2021/08/19 18:15:09 kid1| Accepting HTTP Socket connections at conn8
local=192.168.101.254:8080 remote=[::] FD 14 flags=9
2021/08/19 18:15:09 kid1| Accepting HTTP Socket connections at conn10
local=127.0.0.1:8080 remote=[::] FD 15 flags=9
2021/08/19 18:15:10 kid1| storeLateRelease: released 0 objects
***SNAP***

>> For my 'cache_peer' problem I opened a bug report
>> (https://bugs.squid-cache.org/show_bug.cgi?id=5147). Work in progress.
> 
> I didn’t get it, but I am sure you know what you are doing :)

Despite being "only" a "non-caching web proxy with advanced filtering
capabilities for enhancing privacy", 'privoxy' still threw away up to
15-20% of unnecessary or unwanted ads or internet junk. I wanted to keep it.

> It is good to work together with upstream.

Yep. And the 'squids' are friendly... ;-)

As I read the last answers (really fast reaction!), these messages about
"failed" TCP connections are triggered by the rejected CONNECTs from
'privoxy'. "Squid v4 did not consider rejected CONNECTs a problem worth
marking the peer DEAD for." They fixed this bug and now 'squid 5.1'
does... And "the bad header field is sent by privoxy".

For now, I've disabled 'privoxy' to see how things are going without it.

Best,
Matthias

Re: squid 5.1 gone stable

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 5808 bytes --]

Hello,

> On 19 Aug 2021, at 17:32, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
> 
> Hi,
> 
> please comments below...
> 
> On 19.08.2021 15:57, Michael Tremer wrote:
>> Hello,
>> 
>>> On 18 Aug 2021, at 17:42, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
>>> 
>>> Hi,
>>> 
>>> On 16.08.2021 11:40, Michael Tremer wrote:
>>> ...
>>>>> What makes me wonder: during build, 'squid' says it can open '32768',
>>>>> during start its '4096'. If someone knows why, please enlighten me... ;-)
>>>> 4096 is the default maximum number of files any process can open at the same.
>>>> 
>>>> This is to protect the system from going crazy by having too many open files (because I think the file descriptor table used to be of a static size in older versions of the kernel).
>>>> 
>>>>>> I suppose this is enough and I can live with 32k. We should remove the field from the UI then.
>>>>> Me too, but are 4096 enough?
>>>> No. I don’t know why the squid team isn’t handling this better. We are hitting this problem every time we update to a new version.
>>>> 
>>>> I suppose this is fine for testing.
>>>> 
>>>> You can try adding “ulimit -n 32768” to the squid init script and then it should be able to open up to 32k files.
>>> ...
>>> 
>>> Thanks for the clarification - I tested this with 'squid 5.1'. It seems
>>> to work:
>>> 
>>> ...
>>> case "1$" in
>>> 	start)
>>> 	ulimit -n 32768
>>> 	getpids "squid"
>>> ...
>> 
>> What is “getpids” good for?
> 
> Its a function call from '/etc/init.d/functions', but please don't ask
> me what its good for: ;-)
> 
> ***SNIP***
> # This will ensure compatibility with previous LFS Bootscripts
> getpids()
> {
>   if [ -z "${PIDFILE}" ]; then
>      pidofproc -s -p "${PIDFILE}" $@
>   else
>     pidofproc -s $@
>   fi
>   base="${1##*/}"
> }
> ***SNAP***

It simply seems to print all PIDs of the squid processes.

>> Adding the ulimit call to the initscript and removing the configuration option from the CGI script is fine with me.
> 
> I'm testing. Works.
> 
> ***SNIP***
> 2021/08/19 18:14:58 kid1| Logfile: closing log
> stdio:/var/log/squid/access.log
> 2021/08/19 18:14:58 kid1| Open FD READ/WRITE 8 redirect_wrapper #1
> 2021/08/19 18:14:58 kid1| Open FD READ/WRITE 10 redirect_wrapper #2
> 2021/08/19 18:14:58 kid1| Squid Cache (Version 5.1): Exiting normally.
> 2021/08/19 18:14:58| Removing PID file (/var/run/squid.pid)
> 2021/08/19 18:15:08 kid1| Current Directory is /srv/web/ipfire/cgi-bin
> 2021/08/19 18:15:08 kid1| Creating missing swap directories
> 2021/08/19 18:15:08 kid1| No cache_dir stores are configured.
> 2021/08/19 18:15:08| Removing PID file (/var/run/squid.pid)
> 2021/08/19 18:15:09 kid1| Current Directory is /srv/web/ipfire/cgi-bin
> 2021/08/19 18:15:09 kid1| Starting Squid Cache version 5.1 for
> x86_64-pc-linux-gnu...
> 2021/08/19 18:15:09 kid1| Service Name: squid
> 2021/08/19 18:15:09 kid1| Process ID 27102
> 2021/08/19 18:15:09 kid1| Process Roles: worker
> 2021/08/19 18:15:09 kid1| With 32768 file descriptors available
> 2021/08/19 18:15:09 kid1| Initializing IP Cache...
> 2021/08/19 18:15:09 kid1| DNS Socket created at 0.0.0.0, FD 7
> 2021/08/19 18:15:09 kid1| Adding domain localdomain from /etc/resolv.conf
> 2021/08/19 18:15:09 kid1| Adding nameserver 127.0.0.1 from /etc/resolv.conf
> 2021/08/19 18:15:09 kid1| helperOpenServers: Starting 2/2
> 'redirect_wrapper' processes
> 2021/08/19 18:15:09 kid1| Logfile: opening log
> stdio:/var/log/squid/access.log
> 2021/08/19 18:15:09 kid1| Store logging disabled
> 2021/08/19 18:15:09 kid1| Swap maxSize 0 + 262144 KB, estimated 20164
> objects
> 2021/08/19 18:15:09 kid1| Target number of buckets: 1008
> 2021/08/19 18:15:09 kid1| Using 8192 Store buckets
> 2021/08/19 18:15:09 kid1| Max Mem size: 262144 KB
> 2021/08/19 18:15:09 kid1| Max Swap size: 0 KB
> 2021/08/19 18:15:09 kid1| Using Least Load store dir selection
> 2021/08/19 18:15:09 kid1| Current Directory is /srv/web/ipfire/cgi-bin
> 2021/08/19 18:15:09 kid1| Finished loading MIME types and icons.
> 2021/08/19 18:15:09 kid1| HTCP Disabled.
> 2021/08/19 18:15:09 kid1| Squid plugin modules loaded: 0
> 2021/08/19 18:15:09 kid1| Adaptation support is off.
> 2021/08/19 18:15:09 kid1| Accepting HTTP Socket connections at conn6
> local=192.168.100.254:8080 remote=[::] FD 13 flags=9
> 2021/08/19 18:15:09 kid1| Accepting HTTP Socket connections at conn8
> local=192.168.101.254:8080 remote=[::] FD 14 flags=9
> 2021/08/19 18:15:09 kid1| Accepting HTTP Socket connections at conn10
> local=127.0.0.1:8080 remote=[::] FD 15 flags=9
> 2021/08/19 18:15:10 kid1| storeLateRelease: released 0 objects
> ***SNAP***
> 
>>> For my 'cache_peer' problem I opened a bug report
>>> (https://bugs.squid-cache.org/show_bug.cgi?id=5147). Work in progress.
>> 
>> I didn’t get it, but I am sure you know what you are doing :)
> 
> Despite being "only" a "non-caching web proxy with advanced filtering
> capabilities for enhancing privacy", 'privoxy' still threw away up to
> 15-20% of unnecessary or unwanted ads or internet junk. I wanted to keep it.

Okay. Does that still work with all this HTTPS traffic?

>> It is good to work together with upstream.
> 
> Yep. And the 'squids' are friendly... ;-)
> 
> As I read the last answers (really fast reaction!), these messages about
> "failed" TCP connections are triggered by the rejected CONNECTs from
> 'privoxy'. "Squid v4 did not consider rejected CONNECTs a problem worth
> marking the peer DEAD for." They fixed this bug and now 'squid 5.1'
> does... And "the bad header field is sent by privoxy".
> 
> For now, I've disabled 'privoxy' to see how things are going without it.
> 
> Best,
> Matthias

Re: squid 5.1 gone stable

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 6589 bytes --]

Hi,

On 23.08.2021 15:29, Michael Tremer wrote:
> Hello,
> 
>> On 19 Aug 2021, at 17:32, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
>> 
>> Hi,
>> 
>> please comments below...
>> 
>> On 19.08.2021 15:57, Michael Tremer wrote:
>>> Hello,
>>> 
>>>> On 18 Aug 2021, at 17:42, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> On 16.08.2021 11:40, Michael Tremer wrote:
>>>> ...
>>>>>> What makes me wonder: during build, 'squid' says it can open '32768',
>>>>>> during start its '4096'. If someone knows why, please enlighten me... ;-)
>>>>> 4096 is the default maximum number of files any process can open at the same.
>>>>> 
>>>>> This is to protect the system from going crazy by having too many open files (because I think the file descriptor table used to be of a static size in older versions of the kernel).
>>>>> 
>>>>>>> I suppose this is enough and I can live with 32k. We should remove the field from the UI then.
>>>>>> Me too, but are 4096 enough?
>>>>> No. I don’t know why the squid team isn’t handling this better. We are hitting this problem every time we update to a new version.
>>>>> 
>>>>> I suppose this is fine for testing.
>>>>> 
>>>>> You can try adding “ulimit -n 32768” to the squid init script and then it should be able to open up to 32k files.
>>>> ...
>>>> 
>>>> Thanks for the clarification - I tested this with 'squid 5.1'. It seems
>>>> to work:
>>>> 
>>>> ...
>>>> case "1$" in
>>>> 	start)
>>>> 	ulimit -n 32768
>>>> 	getpids "squid"
>>>> ...
>>> 
>>> What is “getpids” good for?
>> 
>> Its a function call from '/etc/init.d/functions', but please don't ask
>> me what its good for: ;-)
>> 
>> ***SNIP***
>> # This will ensure compatibility with previous LFS Bootscripts
>> getpids()
>> {
>>   if [ -z "${PIDFILE}" ]; then
>>      pidofproc -s -p "${PIDFILE}" $@
>>   else
>>     pidofproc -s $@
>>   fi
>>   base="${1##*/}"
>> }
>> ***SNAP***
> 
> It simply seems to print all PIDs of the squid processes.

Thanks. I thought something like that but wasn't sure.

>>> Adding the ulimit call to the initscript and removing the configuration option from the CGI script is fine with me.
>> 
>> I'm testing. Works.
>> 
>> ***SNIP***
>> 2021/08/19 18:14:58 kid1| Logfile: closing log
>> stdio:/var/log/squid/access.log
>> 2021/08/19 18:14:58 kid1| Open FD READ/WRITE 8 redirect_wrapper #1
>> 2021/08/19 18:14:58 kid1| Open FD READ/WRITE 10 redirect_wrapper #2
>> 2021/08/19 18:14:58 kid1| Squid Cache (Version 5.1): Exiting normally.
>> 2021/08/19 18:14:58| Removing PID file (/var/run/squid.pid)
>> 2021/08/19 18:15:08 kid1| Current Directory is /srv/web/ipfire/cgi-bin
>> 2021/08/19 18:15:08 kid1| Creating missing swap directories
>> 2021/08/19 18:15:08 kid1| No cache_dir stores are configured.
>> 2021/08/19 18:15:08| Removing PID file (/var/run/squid.pid)
>> 2021/08/19 18:15:09 kid1| Current Directory is /srv/web/ipfire/cgi-bin
>> 2021/08/19 18:15:09 kid1| Starting Squid Cache version 5.1 for
>> x86_64-pc-linux-gnu...
>> 2021/08/19 18:15:09 kid1| Service Name: squid
>> 2021/08/19 18:15:09 kid1| Process ID 27102
>> 2021/08/19 18:15:09 kid1| Process Roles: worker
>> 2021/08/19 18:15:09 kid1| With 32768 file descriptors available
>> 2021/08/19 18:15:09 kid1| Initializing IP Cache...
>> 2021/08/19 18:15:09 kid1| DNS Socket created at 0.0.0.0, FD 7
>> 2021/08/19 18:15:09 kid1| Adding domain localdomain from /etc/resolv.conf
>> 2021/08/19 18:15:09 kid1| Adding nameserver 127.0.0.1 from /etc/resolv.conf
>> 2021/08/19 18:15:09 kid1| helperOpenServers: Starting 2/2
>> 'redirect_wrapper' processes
>> 2021/08/19 18:15:09 kid1| Logfile: opening log
>> stdio:/var/log/squid/access.log
>> 2021/08/19 18:15:09 kid1| Store logging disabled
>> 2021/08/19 18:15:09 kid1| Swap maxSize 0 + 262144 KB, estimated 20164
>> objects
>> 2021/08/19 18:15:09 kid1| Target number of buckets: 1008
>> 2021/08/19 18:15:09 kid1| Using 8192 Store buckets
>> 2021/08/19 18:15:09 kid1| Max Mem size: 262144 KB
>> 2021/08/19 18:15:09 kid1| Max Swap size: 0 KB
>> 2021/08/19 18:15:09 kid1| Using Least Load store dir selection
>> 2021/08/19 18:15:09 kid1| Current Directory is /srv/web/ipfire/cgi-bin
>> 2021/08/19 18:15:09 kid1| Finished loading MIME types and icons.
>> 2021/08/19 18:15:09 kid1| HTCP Disabled.
>> 2021/08/19 18:15:09 kid1| Squid plugin modules loaded: 0
>> 2021/08/19 18:15:09 kid1| Adaptation support is off.
>> 2021/08/19 18:15:09 kid1| Accepting HTTP Socket connections at conn6
>> local=192.168.100.254:8080 remote=[::] FD 13 flags=9
>> 2021/08/19 18:15:09 kid1| Accepting HTTP Socket connections at conn8
>> local=192.168.101.254:8080 remote=[::] FD 14 flags=9
>> 2021/08/19 18:15:09 kid1| Accepting HTTP Socket connections at conn10
>> local=127.0.0.1:8080 remote=[::] FD 15 flags=9
>> 2021/08/19 18:15:10 kid1| storeLateRelease: released 0 objects
>> ***SNAP***
>> 
>>>> For my 'cache_peer' problem I opened a bug report
>>>> (https://bugs.squid-cache.org/show_bug.cgi?id=5147). Work in progress.
>>> 
>>> I didn’t get it, but I am sure you know what you are doing :)
>> 
>> Despite being "only" a "non-caching web proxy with advanced filtering
>> capabilities for enhancing privacy", 'privoxy' still threw away up to
>> 15-20% of unnecessary or unwanted ads or internet junk. I wanted to keep it.
> 
> Okay. Does that still work with all this HTTPS traffic?

I had no problems with https. Never. Only the children sometimes
complained... ;-)
If you can trust the privoxy statistics, it is surprising how much
banners, ads, junk, webbugs, popups etc. are still filtered out. I added
some common blocklists (e.g, Fanboy, Malware, Easylist) for this. With
the last version I started testing it with 'brotli 1.0.9' support.
Worked for me. But as said, 'squid 5.1' has its problems with 'privoxy'.
I'm at it - right now in "privoxy-users(a)lists.privoxy.org".

Best,
Matthias

>>> It is good to work together with upstream.
>> 
>> Yep. And the 'squids' are friendly... ;-)
>> 
>> As I read the last answers (really fast reaction!), these messages about
>> "failed" TCP connections are triggered by the rejected CONNECTs from
>> 'privoxy'. "Squid v4 did not consider rejected CONNECTs a problem worth
>> marking the peer DEAD for." They fixed this bug and now 'squid 5.1'
>> does... And "the bad header field is sent by privoxy".
>> 
>> For now, I've disabled 'privoxy' to see how things are going without it.
>> 
>> Best,
>> Matthias
> 

squid 5.1 gone stable

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 571 bytes --]

Hi,

for the records...

5.1 has gone stable:

=> http://www.squid-cache.org/Versions/

Excerpt from changelog:
"Changes in squid-5.1 (01 Aug 2021):

	- Bug 4696: Fix leaky String move assignment operator
	- Fix ACL-related reconfiguration memory leak
	- Fix SSL-Bump reconfiguration leaking public key memory
	- Fix build on RISC-V
	- Fix build on Ubuntu 21.04

Changes in squid-5.0.7 (04 Jul 2021):

	- Fix a helper logging issues
	- Fix some helper connection issues
	- Cleanup: remove much unused code
	- ... and all fixes from 4.16
..."

I'm at it.

Best,
Matthias

Re: squid 5.1 gone stable

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 995 bytes --]

Hey,

That sounds good.

The change log lists a couple of changed and remove directives:

  http://www.squid-cache.org/Versions/v5/squid-5.1-RELEASENOTES.html#s3

Could you check if we are affected by any of this change?

-Michael

> On 2 Aug 2021, at 17:12, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
> 
> Hi,
> 
> for the records...
> 
> 5.1 has gone stable:
> 
> => http://www.squid-cache.org/Versions/
> 
> Excerpt from changelog:
> "Changes in squid-5.1 (01 Aug 2021):
> 
> 	- Bug 4696: Fix leaky String move assignment operator
> 	- Fix ACL-related reconfiguration memory leak
> 	- Fix SSL-Bump reconfiguration leaking public key memory
> 	- Fix build on RISC-V
> 	- Fix build on Ubuntu 21.04
> 
> Changes in squid-5.0.7 (04 Jul 2021):
> 
> 	- Fix a helper logging issues
> 	- Fix some helper connection issues
> 	- Cleanup: remove much unused code
> 	- ... and all fixes from 4.16
> ..."
> 
> I'm at it.
> 
> Best,
> Matthias
> 

Re: squid 5.1 gone stable

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 2824 bytes --]

Hi,

On 02.08.2021 18:39, Michael Tremer wrote:
> Hey,
> 
> That sounds good.
> 
> The change log lists a couple of changed and remove directives:
> 
>   http://www.squid-cache.org/Versions/v5/squid-5.1-RELEASENOTES.html#s3
> 
> Could you check if we are affected by any of this change?

Building was ok - installing was ok. I checked the release notes and on
the first glance didn't find anything we would need to worry about. Will
take a closer look on weekend.

But...:

5.1 is permanently complaining about filedescriptors and I couldn't find
the reason for this so far. Even if I reduce the number during build to
32768 (this is supposedly his maximum), it keeps complaining:

...
2021/08/03 19:19:03 kid1| NOTICE: Could not increase the number of
filedescriptors
2021/08/03 19:19:03 kid1| With 4096 file descriptors available
...

And - it seems that it doesn't like 'privoxy'. Everything seems to
work(!), no seen problems(!), but with 'squid 5.1' I'm getting these
sporadic messages in 'cache_log':

...
2021/08/03 19:27:17 kid1| helperOpenServers: Starting 2/4
'redirect_wrapper' processes
    current master transaction: master88
2021/08/03 19:27:33 kid1| TCP connection to 127.0.0.1/8118 failed
    current master transaction: master53
2021/08/03 19:27:34 kid1| TCP connection to 127.0.0.1/8118 failed
    current master transaction: master53
2021/08/03 19:27:56 kid1| TCP connection to 127.0.0.1/8118 failed
    current master transaction: master53
...
2021/08/03 19:59:34 kid1| TCP connection to 127.0.0.1/8118 failed
    current master transaction: master53
2021/08/03 19:59:34 kid1| Detected DEAD Parent: privoxy
    current master transaction: master53
2021/08/03 19:59:34 kid1| Detected REVIVED Parent: privoxy
    current master transaction: master53
2021/08/03 19:59:34 kid1| TCP connection to 127.0.0.1/8118 failed
    current master transaction: master53
...

Somehow I don't like this. Will report, if I find the reason.

Best,
Matthias


> -Michael
> 
>> On 2 Aug 2021, at 17:12, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
>> 
>> Hi,
>> 
>> for the records...
>> 
>> 5.1 has gone stable:
>> 
>> => http://www.squid-cache.org/Versions/
>> 
>> Excerpt from changelog:
>> "Changes in squid-5.1 (01 Aug 2021):
>> 
>> 	- Bug 4696: Fix leaky String move assignment operator
>> 	- Fix ACL-related reconfiguration memory leak
>> 	- Fix SSL-Bump reconfiguration leaking public key memory
>> 	- Fix build on RISC-V
>> 	- Fix build on Ubuntu 21.04
>> 
>> Changes in squid-5.0.7 (04 Jul 2021):
>> 
>> 	- Fix a helper logging issues
>> 	- Fix some helper connection issues
>> 	- Cleanup: remove much unused code
>> 	- ... and all fixes from 4.16
>> ..."
>> 
>> I'm at it.
>> 
>> Best,
>> Matthias
>> 
> 

Re: squid 5.1 gone stable

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 3258 bytes --]

Hi,

> On 5 Aug 2021, at 19:12, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
> 
> Hi,
> 
> On 02.08.2021 18:39, Michael Tremer wrote:
>> Hey,
>> 
>> That sounds good.
>> 
>> The change log lists a couple of changed and remove directives:
>> 
>>  http://www.squid-cache.org/Versions/v5/squid-5.1-RELEASENOTES.html#s3
>> 
>> Could you check if we are affected by any of this change?
> 
> Building was ok - installing was ok. I checked the release notes and on
> the first glance didn't find anything we would need to worry about. Will
> take a closer look on weekend.
> 
> But...:
> 
> 5.1 is permanently complaining about filedescriptors and I couldn't find
> the reason for this so far. Even if I reduce the number during build to
> 32768 (this is supposedly his maximum), it keeps complaining:
> 
> ...
> 2021/08/03 19:19:03 kid1| NOTICE: Could not increase the number of
> filedescriptors
> 2021/08/03 19:19:03 kid1| With 4096 file descriptors available
> ...

That must be the system’s limit then. If the process is being launched as root, it should be able to change its own limits.

> And - it seems that it doesn't like 'privoxy'. Everything seems to
> work(!), no seen problems(!), but with 'squid 5.1' I'm getting these
> sporadic messages in 'cache_log':

We don’t have privoxy. Is that a custom thing that you are running?

> ...
> 2021/08/03 19:27:17 kid1| helperOpenServers: Starting 2/4
> 'redirect_wrapper' processes
>    current master transaction: master88
> 2021/08/03 19:27:33 kid1| TCP connection to 127.0.0.1/8118 failed
>    current master transaction: master53
> 2021/08/03 19:27:34 kid1| TCP connection to 127.0.0.1/8118 failed
>    current master transaction: master53
> 2021/08/03 19:27:56 kid1| TCP connection to 127.0.0.1/8118 failed
>    current master transaction: master53
> ...
> 2021/08/03 19:59:34 kid1| TCP connection to 127.0.0.1/8118 failed
>    current master transaction: master53
> 2021/08/03 19:59:34 kid1| Detected DEAD Parent: privoxy
>    current master transaction: master53
> 2021/08/03 19:59:34 kid1| Detected REVIVED Parent: privoxy
>    current master transaction: master53
> 2021/08/03 19:59:34 kid1| TCP connection to 127.0.0.1/8118 failed
>    current master transaction: master53
> ...
> 
> Somehow I don't like this. Will report, if I find the reason.
> 
> Best,
> Matthias
> 
> 
>> -Michael
>> 
>>> On 2 Aug 2021, at 17:12, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
>>> 
>>> Hi,
>>> 
>>> for the records...
>>> 
>>> 5.1 has gone stable:
>>> 
>>> => http://www.squid-cache.org/Versions/
>>> 
>>> Excerpt from changelog:
>>> "Changes in squid-5.1 (01 Aug 2021):
>>> 
>>> 	- Bug 4696: Fix leaky String move assignment operator
>>> 	- Fix ACL-related reconfiguration memory leak
>>> 	- Fix SSL-Bump reconfiguration leaking public key memory
>>> 	- Fix build on RISC-V
>>> 	- Fix build on Ubuntu 21.04
>>> 
>>> Changes in squid-5.0.7 (04 Jul 2021):
>>> 
>>> 	- Fix a helper logging issues
>>> 	- Fix some helper connection issues
>>> 	- Cleanup: remove much unused code
>>> 	- ... and all fixes from 4.16
>>> ..."
>>> 
>>> I'm at it.
>>> 
>>> Best,
>>> Matthias
>>> 
>> 
> 

Re: squid 5.1 gone stable

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 3969 bytes --]

Hi,

On 06.08.2021 12:46, Michael Tremer wrote:
> Hi,
> 
>> On 5 Aug 2021, at 19:12, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
>> 
>> Hi,
>> 
>> On 02.08.2021 18:39, Michael Tremer wrote:
>>> Hey,
>>> 
>>> That sounds good.
>>> 
>>> The change log lists a couple of changed and remove directives:
>>> 
>>>  http://www.squid-cache.org/Versions/v5/squid-5.1-RELEASENOTES.html#s3
>>> 
>>> Could you check if we are affected by any of this change?
>> 
>> Building was ok - installing was ok. I checked the release notes and on
>> the first glance didn't find anything we would need to worry about. Will
>> take a closer look on weekend.
>> 
>> But...:
>> 
>> 5.1 is permanently complaining about filedescriptors and I couldn't find
>> the reason for this so far. Even if I reduce the number during build to
>> 32768 (this is supposedly his maximum), it keeps complaining:
>> 
>> ...
>> 2021/08/03 19:19:03 kid1| NOTICE: Could not increase the number of
>> filedescriptors
>> 2021/08/03 19:19:03 kid1| With 4096 file descriptors available
>> ...
> 
> That must be the system’s limit then. If the process is being launched as root, it should be able to change its own limits.

Yes. I think so. Its weird. 'squid 4.16' - running with *exactly* the
same configuration (under Core 158) - doesn't complain. Only '5.1'
won't. I'm searching.

>> And - it seems that it doesn't like 'privoxy'. Everything seems to
>> work(!), no seen problems(!), but with 'squid 5.1' I'm getting these
>> sporadic messages in 'cache_log':
> 
> We don’t have privoxy. Is that a custom thing that you are running?

Yes.
That one leftover from the old copfilter, ported to IFire. We once
talked about this... I rewrote it as an addon for IPFire. Its running
without any problems with the last 'squid 4.*' versions so far. But as
above, 'squid 5.1' somehow doesn't like it anymore. 'privoxy' is still
filtering and working, but I'm getting these complains from 'squid'. Hm.

Best,
Matthias

>> ...
>> 2021/08/03 19:27:17 kid1| helperOpenServers: Starting 2/4
>> 'redirect_wrapper' processes
>>    current master transaction: master88
>> 2021/08/03 19:27:33 kid1| TCP connection to 127.0.0.1/8118 failed
>>    current master transaction: master53
>> 2021/08/03 19:27:34 kid1| TCP connection to 127.0.0.1/8118 failed
>>    current master transaction: master53
>> 2021/08/03 19:27:56 kid1| TCP connection to 127.0.0.1/8118 failed
>>    current master transaction: master53
>> ...
>> 2021/08/03 19:59:34 kid1| TCP connection to 127.0.0.1/8118 failed
>>    current master transaction: master53
>> 2021/08/03 19:59:34 kid1| Detected DEAD Parent: privoxy
>>    current master transaction: master53
>> 2021/08/03 19:59:34 kid1| Detected REVIVED Parent: privoxy
>>    current master transaction: master53
>> 2021/08/03 19:59:34 kid1| TCP connection to 127.0.0.1/8118 failed
>>    current master transaction: master53
>> ...
>> 
>> Somehow I don't like this. Will report, if I find the reason.
>> 
>> Best,
>> Matthias
>> 
>> 
>>> -Michael
>>> 
>>>> On 2 Aug 2021, at 17:12, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> for the records...
>>>> 
>>>> 5.1 has gone stable:
>>>> 
>>>> => http://www.squid-cache.org/Versions/
>>>> 
>>>> Excerpt from changelog:
>>>> "Changes in squid-5.1 (01 Aug 2021):
>>>> 
>>>> 	- Bug 4696: Fix leaky String move assignment operator
>>>> 	- Fix ACL-related reconfiguration memory leak
>>>> 	- Fix SSL-Bump reconfiguration leaking public key memory
>>>> 	- Fix build on RISC-V
>>>> 	- Fix build on Ubuntu 21.04
>>>> 
>>>> Changes in squid-5.0.7 (04 Jul 2021):
>>>> 
>>>> 	- Fix a helper logging issues
>>>> 	- Fix some helper connection issues
>>>> 	- Cleanup: remove much unused code
>>>> 	- ... and all fixes from 4.16
>>>> ..."
>>>> 
>>>> I'm at it.
>>>> 
>>>> Best,
>>>> Matthias
>>>> 
>>> 
>> 
> 

Re: squid 5.1 gone stable

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 10382 bytes --]

Hi,

For the records...

I made a few start/restart tests with '4.16' and '5.1' today.

What is puzzling me => please pay attention to the filedescriptor messages:

'squid 4.16' - cache_log:
***SNIP***
2021/08/07 01:27:23 kid1| Squid Cache (Version 4.16): Exiting normally.
2021/08/07 01:27:23| Removing PID file (/var/run/squid.pid)
2021/08/07 01:27:23| Current Directory is /
2021/08/07 01:27:23| Current Directory is /
2021/08/07 01:27:35| Created PID file (/var/run/squid.pid)
2021/08/07 01:27:35 kid1| Current Directory is /
2021/08/07 01:27:35 kid1| Creating missing swap directories
2021/08/07 01:27:35 kid1| No cache_dir stores are configured.
2021/08/07 01:27:35| Removing PID file (/var/run/squid.pid)
2021/08/07 01:27:36| Created PID file (/var/run/squid.pid)
2021/08/07 01:27:36 kid1| Current Directory is /
2021/08/07 01:27:36 kid1| Starting Squid Cache version 4.16 for
x86_64-pc-linux-gnu...
2021/08/07 01:27:36 kid1| Service Name: squid
2021/08/07 01:27:36 kid1| Process ID 14593
2021/08/07 01:27:36 kid1| Process Roles: worker
*2021/08/07 01:27:36 kid1| With 16384 file descriptors available* <----!
2021/08/07 01:27:36 kid1| Initializing IP Cache...
2021/08/07 01:27:36 kid1| DNS Socket created at 0.0.0.0, FD 5
2021/08/07 01:27:36 kid1| Adding domain localdomain from /etc/resolv.conf
2021/08/07 01:27:36 kid1| Adding nameserver 127.0.0.1 from /etc/resolv.conf
2021/08/07 01:27:36 kid1| helperOpenServers: Starting 2/4
'redirect_wrapper' processes
2021/08/07 01:27:36 kid1| Logfile: opening log
stdio:/var/log/squid/access.log
2021/08/07 01:27:36 kid1| Store logging disabled
2021/08/07 01:27:36 kid1| Swap maxSize 0 + 262144 KB, estimated 20164
objects
2021/08/07 01:27:36 kid1| Target number of buckets: 1008
2021/08/07 01:27:36 kid1| Using 8192 Store buckets
2021/08/07 01:27:36 kid1| Max Mem size: 262144 KB
2021/08/07 01:27:36 kid1| Max Swap size: 0 KB
2021/08/07 01:27:36 kid1| Using Least Load store dir selection
2021/08/07 01:27:36 kid1| Current Directory is /
2021/08/07 01:27:36 kid1| Finished loading MIME types and icons.
2021/08/07 01:27:36 kid1| HTCP Disabled.
2021/08/07 01:27:36 kid1| Configuring Parent 127.0.0.1/8118/7
2021/08/07 01:27:36 kid1| Squid plugin modules loaded: 0
2021/08/07 01:27:36 kid1| Adaptation support is off.
2021/08/07 01:27:36 kid1| Accepting HTTP Socket connections at
local=192.168.100.254:8080 remote=[::] FD 13 flags=9
2021/08/07 01:27:36 kid1| Accepting HTTP Socket connections at
local=192.168.101.254:8080 remote=[::] FD 14 flags=9
2021/08/07 01:27:36 kid1| Accepting HTTP Socket connections at
local=127.0.0.1:8080 remote=[::] FD 15 flags=9
2021/08/07 01:27:36 kid1| Starting new redirector helpers...
2021/08/07 01:27:36 kid1| helperOpenServers: Starting 2/4
'redirect_wrapper' processes
2021/08/07 01:27:37 kid1| storeLateRelease: released 0 objects
***SNAP***

'squid 5.1 - cache_log - with *exactly* the same configuration:
***SNIP***
2021/08/07 09:25:29 kid1| Squid Cache (Version 4.16): Exiting normally.
2021/08/07 09:25:29| Removing PID file (/var/run/squid.pid)
2021/08/07 09:25:29| Current Directory is /
2021/08/07 09:25:29| Current Directory is /
2021/08/07 09:25:46 kid1| Current Directory is /
2021/08/07 09:25:46 kid1| Creating missing swap directories
2021/08/07 09:25:46 kid1| No cache_dir stores are configured.
2021/08/07 09:25:46| Removing PID file (/var/run/squid.pid)
2021/08/07 09:25:46 kid1| Current Directory is /
2021/08/07 09:25:46 kid1| Starting Squid Cache version 5.1 for
x86_64-pc-linux-gnu...
2021/08/07 09:25:46 kid1| Service Name: squid
2021/08/07 09:25:46 kid1| Process ID 8881
2021/08/07 09:25:46 kid1| Process Roles: worker
*2021/08/07 09:25:46 kid1| NOTICE: Could not increase the number of
filedescriptors*              <------------!!!!!
*2021/08/07 09:25:46 kid1| With 4096 file descriptors available*
2021/08/07 09:25:46 kid1| Initializing IP Cache...
2021/08/07 09:25:46 kid1| DNS Socket created at 0.0.0.0, FD 7
2021/08/07 09:25:46 kid1| Adding domain localdomain from /etc/resolv.conf
2021/08/07 09:25:46 kid1| Adding nameserver 127.0.0.1 from /etc/resolv.conf
2021/08/07 09:25:46 kid1| helperOpenServers: Starting 2/4
'redirect_wrapper' processes
2021/08/07 09:25:46 kid1| Logfile: opening log
stdio:/var/log/squid/access.log
2021/08/07 09:25:46 kid1| Store logging disabled
2021/08/07 09:25:46 kid1| Swap maxSize 0 + 262144 KB, estimated 20164
objects
2021/08/07 09:25:46 kid1| Target number of buckets: 1008
2021/08/07 09:25:46 kid1| Using 8192 Store buckets
2021/08/07 09:25:46 kid1| Max Mem size: 262144 KB
2021/08/07 09:25:46 kid1| Max Swap size: 0 KB
2021/08/07 09:25:46 kid1| Using Least Load store dir selection
2021/08/07 09:25:46 kid1| Current Directory is /
2021/08/07 09:25:46 kid1| Finished loading MIME types and icons.
2021/08/07 09:25:46 kid1| HTCP Disabled.
2021/08/07 09:25:46 kid1| Configuring Parent 127.0.0.1/8118/7
2021/08/07 09:25:46 kid1| Squid plugin modules loaded: 0
2021/08/07 09:25:46 kid1| Adaptation support is off.
2021/08/07 09:25:46 kid1| Accepting HTTP Socket connections at conn6
local=192.168.100.254:8080 remote=[::] FD 13 flags=9
2021/08/07 09:25:46 kid1| Accepting HTTP Socket connections at conn8
local=192.168.101.254:8080 remote=[::] FD 14 flags=9
2021/08/07 09:25:46 kid1| Accepting HTTP Socket connections at conn10
local=127.0.0.1:8080 remote=[::] FD 15 flags=9
2021/08/07 09:25:47 kid1| storeLateRelease: released 0 objects
*2021/08/07 09:27:33 kid1| TCP connection to 127.0.0.1/8118 failed
current master transaction: master53*  <----- thats 'privoxy'
2021/08/07 09:28:34 kid1| Starting new redirector helpers...
...

'privoxy' is another problem - besides, no errors can be detected during
browsing/operation. Everything seems to be ok. I just get these warnings
in 'cache_log'.

For 'privoxy' to work I had to add these lines in 'squid.conf', which -
until now - never made any problems with 4.x versions:

...
cache_peer 127.0.0.1 parent 8118 0 name=privoxy no-query no-digest
no-netdb-exchange default
never_direct allow all
...

If "someone" has any ideas what is causing this:
Hints are welcome... ;-)

Best,
Matthias


On 06.08.2021 19:55, Matthias Fischer wrote:
> Hi,
> 
> On 06.08.2021 12:46, Michael Tremer wrote:
>> Hi,
>> 
>>> On 5 Aug 2021, at 19:12, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
>>> 
>>> Hi,
>>> 
>>> On 02.08.2021 18:39, Michael Tremer wrote:
>>>> Hey,
>>>> 
>>>> That sounds good.
>>>> 
>>>> The change log lists a couple of changed and remove directives:
>>>> 
>>>>  http://www.squid-cache.org/Versions/v5/squid-5.1-RELEASENOTES.html#s3
>>>> 
>>>> Could you check if we are affected by any of this change?
>>> 
>>> Building was ok - installing was ok. I checked the release notes and on
>>> the first glance didn't find anything we would need to worry about. Will
>>> take a closer look on weekend.
>>> 
>>> But...:
>>> 
>>> 5.1 is permanently complaining about filedescriptors and I couldn't find
>>> the reason for this so far. Even if I reduce the number during build to
>>> 32768 (this is supposedly his maximum), it keeps complaining:
>>> 
>>> ...
>>> 2021/08/03 19:19:03 kid1| NOTICE: Could not increase the number of
>>> filedescriptors
>>> 2021/08/03 19:19:03 kid1| With 4096 file descriptors available
>>> ...
>> 
>> That must be the system’s limit then. If the process is being launched as root, it should be able to change its own limits.
> 
> Yes. I think so. Its weird. 'squid 4.16' - running with *exactly* the
> same configuration (under Core 158) - doesn't complain. Only '5.1'
> won't. I'm searching.
> 
>>> And - it seems that it doesn't like 'privoxy'. Everything seems to
>>> work(!), no seen problems(!), but with 'squid 5.1' I'm getting these
>>> sporadic messages in 'cache_log':
>> 
>> We don’t have privoxy. Is that a custom thing that you are running?
> 
> Yes.
> That one leftover from the old copfilter, ported to IFire. We once
> talked about this... I rewrote it as an addon for IPFire. Its running
> without any problems with the last 'squid 4.*' versions so far. But as
> above, 'squid 5.1' somehow doesn't like it anymore. 'privoxy' is still
> filtering and working, but I'm getting these complains from 'squid'. Hm.
> 
> Best,
> Matthias
> 
>>> ...
>>> 2021/08/03 19:27:17 kid1| helperOpenServers: Starting 2/4
>>> 'redirect_wrapper' processes
>>>    current master transaction: master88
>>> 2021/08/03 19:27:33 kid1| TCP connection to 127.0.0.1/8118 failed
>>>    current master transaction: master53
>>> 2021/08/03 19:27:34 kid1| TCP connection to 127.0.0.1/8118 failed
>>>    current master transaction: master53
>>> 2021/08/03 19:27:56 kid1| TCP connection to 127.0.0.1/8118 failed
>>>    current master transaction: master53
>>> ...
>>> 2021/08/03 19:59:34 kid1| TCP connection to 127.0.0.1/8118 failed
>>>    current master transaction: master53
>>> 2021/08/03 19:59:34 kid1| Detected DEAD Parent: privoxy
>>>    current master transaction: master53
>>> 2021/08/03 19:59:34 kid1| Detected REVIVED Parent: privoxy
>>>    current master transaction: master53
>>> 2021/08/03 19:59:34 kid1| TCP connection to 127.0.0.1/8118 failed
>>>    current master transaction: master53
>>> ...
>>> 
>>> Somehow I don't like this. Will report, if I find the reason.
>>> 
>>> Best,
>>> Matthias
>>> 
>>> 
>>>> -Michael
>>>> 
>>>>> On 2 Aug 2021, at 17:12, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> for the records...
>>>>> 
>>>>> 5.1 has gone stable:
>>>>> 
>>>>> => http://www.squid-cache.org/Versions/
>>>>> 
>>>>> Excerpt from changelog:
>>>>> "Changes in squid-5.1 (01 Aug 2021):
>>>>> 
>>>>> 	- Bug 4696: Fix leaky String move assignment operator
>>>>> 	- Fix ACL-related reconfiguration memory leak
>>>>> 	- Fix SSL-Bump reconfiguration leaking public key memory
>>>>> 	- Fix build on RISC-V
>>>>> 	- Fix build on Ubuntu 21.04
>>>>> 
>>>>> Changes in squid-5.0.7 (04 Jul 2021):
>>>>> 
>>>>> 	- Fix a helper logging issues
>>>>> 	- Fix some helper connection issues
>>>>> 	- Cleanup: remove much unused code
>>>>> 	- ... and all fixes from 4.16
>>>>> ..."
>>>>> 
>>>>> I'm at it.
>>>>> 
>>>>> Best,
>>>>> Matthias
>>>>> 
>>>> 
>>> 
>> 
>