Merge Window Closing Announcement for Core Update 127

Merge Window Closing Announcement for Core Update 127

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 863 bytes --]

Hello guys,

Upon request, I would like to announce that on Saturday, 12th Jan, we are going to branch Core Update 127.

This means that we no longer accept any patches that introduce any new features and only accept patches that fix any bugs introduced. Nothing new here, you all know how this works :)

This announcement is just coming now so that you all know this. It is a bit short notice this time, but I hope I will be better with this next time round. In the past some patches have been submitted after the merge window closed and had to be deferred to the next Core Update to the frustration of the submitters. Therefore, we now try this and hope that we can make the process better for everyone who is contributing!

We have an update with many new features here. Please help testing as soon as the first build is available.

Best,
-Michael

Re: Merge Window Closing Announcement for Core Update 127

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 1407 bytes --]

Hello Michael,

thank you for the notification. :-)

Currently, I do not have anything for Core Update 127 left
and unfortunately, things are very busy.

I will try to get something done around Suricata, IPFire 3.x
and so on over the weekend.

Thank you for the patience, and best regards,
Peter Müller

> Hello guys,
> 
> Upon request, I would like to announce that on Saturday, 12th Jan, we are going to branch Core Update 127.
> 
> This means that we no longer accept any patches that introduce any new features and only accept patches that fix any bugs introduced. Nothing new here, you all know how this works :)
> 
> This announcement is just coming now so that you all know this. It is a bit short notice this time, but I hope I will be better with this next time round. In the past some patches have been submitted after the merge window closed and had to be deferred to the next Core Update to the frustration of the submitters. Therefore, we now try this and hope that we can make the process better for everyone who is contributing!
> 
> We have an update with many new features here. Please help testing as soon as the first build is available.
> 
> Best,
> -Michael
> 


-- 
Microsoft DNS service terminates abnormally when it recieves a response
to a DNS query that was never made.  Fix Information: Run your DNS
service on a different platform.
		-- bugtraq

Re: Merge Window Closing Announcement for Core Update 127

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 1751 bytes --]

On 10.01.2019 15:22, Michael Tremer wrote:
> Hello guys,

Hi,

> Upon request, I would like to announce that on Saturday, 12th Jan, we are going to branch Core Update 127.
> 
> This means that we no longer accept any patches that introduce any new features and only accept patches that fix any bugs introduced. Nothing new here, you all know how this works :)
> 
> This announcement is just coming now so that you all know this. It is a bit short notice this time, but I hope I will be better with this next time round. In the past some patches have been submitted after the merge window closed and had to be deferred to the next Core Update to the frustration of the submitters. Therefore, we now try this and hope that we can make the process better for everyone who is contributing!
> 
> We have an update with many new features here. Please help testing as soon as the first build is available.
> ...

Thanks for the reminder, but regarding this Core update, one question is
bothering me:

Did anyone had the chance to test 'squid 4.5' - especially along with
this commit
(https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=eedca6e36c1131ce5542da5ccbfbb5667648c024
=> "squid: Run as many redirectors as we have CPU cores") - in a
productive environment?

During my first tests, I got occasional warnings like "idnsSendQuery FD
8: sendto: (1) Operation not permitted" or "WARNING: All 2/2 redirector
processes are busy" in squid 'cache.log' and I'd like to clarify
possible mistakes in advance.

By now, I'm running the untuned parameters from current 'next' and these
errors haven't appeared for some time now. Furthermore, 'testing tree'
exists, ( ;-) ) but did anyone had a chance to try this?

Best,
Matthias

Re: Merge Window Closing Announcement for Core Update 127

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 2041 bytes --]

Hi,

> On 13 Jan 2019, at 10:04, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote:
> 
> On 10.01.2019 15:22, Michael Tremer wrote:
>> Hello guys,
> 
> Hi,
> 
>> Upon request, I would like to announce that on Saturday, 12th Jan, we are going to branch Core Update 127.
>> 
>> This means that we no longer accept any patches that introduce any new features and only accept patches that fix any bugs introduced. Nothing new here, you all know how this works :)
>> 
>> This announcement is just coming now so that you all know this. It is a bit short notice this time, but I hope I will be better with this next time round. In the past some patches have been submitted after the merge window closed and had to be deferred to the next Core Update to the frustration of the submitters. Therefore, we now try this and hope that we can make the process better for everyone who is contributing!
>> 
>> We have an update with many new features here. Please help testing as soon as the first build is available.
>> ...
> 
> Thanks for the reminder, but regarding this Core update, one question is
> bothering me:
> 
> Did anyone had the chance to test 'squid 4.5' - especially along with
> this commit
> (https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=eedca6e36c1131ce5542da5ccbfbb5667648c024
> => "squid: Run as many redirectors as we have CPU cores") - in a
> productive environment?
> 
> During my first tests, I got occasional warnings like "idnsSendQuery FD
> 8: sendto: (1) Operation not permitted" or "WARNING: All 2/2 redirector
> processes are busy" in squid 'cache.log' and I'd like to clarify
> possible mistakes in advance.
> 
> By now, I'm running the untuned parameters from current 'next' and these
> errors haven't appeared for some time now. Furthermore, 'testing tree'
> exists, ( ;-) ) but did anyone had a chance to try this?

I sent squid thousands of requests with ab (Apache Benchmark) and that worked fine with only one redirector.

Best,
-Michael

> 
> Best,
> Matthias

Re: Merge Window Closing Announcement for Core Update 127

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 205 bytes --]

Hi,

On 13.01.2019 13:14, Michael Tremer wrote:
> I sent squid thousands of requests with ab (Apache Benchmark) and that worked fine with only one redirector.

Thanks for the feedback! Merged. ;-)

Aw: Re: Merge Window Closing Announcement for Core Update 127

[Bernhard Bitsch]

[-- Attachment #1: Type: text/plain, Size: 869 bytes --]



> Gesendet: Sonntag, 13. Januar 2019 um 13:43 Uhr
> Von: "Matthias Fischer" <matthias.fischer(a)ipfire.org>
> An: "Michael Tremer" <michael.tremer(a)ipfire.org>
> Cc: "IPFire: Development-List" <development(a)lists.ipfire.org>
> Betreff: Re: Merge Window Closing Announcement for Core Update 127
>
> Hi,
> 
> On 13.01.2019 13:14, Michael Tremer wrote:
> > I sent squid thousands of requests with ab (Apache Benchmark) and that worked fine with only one redirector.
> 
> Thanks for the feedback! Merged. ;-)
> 
> 

Nevertheless it's worth reasoning about multiprocessing ( #redirectors = #processors ) versus multitasking ( #redirectors = #task needed by the environment ).
Even with only one processor it is possible to serve multiple requests. Using wait phases for other processes is a basic feature of multitasking.

- Bernhard ( aka BeBiMa )

Re: Merge Window Closing Announcement for Core Update 127

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1619 bytes --]

Hello,

> On 13 Jan 2019, at 14:24, Bernhard Bitsch <Bernhard.Bitsch(a)gmx.de> wrote:
> 
> 
> 
>> Gesendet: Sonntag, 13. Januar 2019 um 13:43 Uhr
>> Von: "Matthias Fischer" <matthias.fischer(a)ipfire.org>
>> An: "Michael Tremer" <michael.tremer(a)ipfire.org>
>> Cc: "IPFire: Development-List" <development(a)lists.ipfire.org>
>> Betreff: Re: Merge Window Closing Announcement for Core Update 127
>> 
>> Hi,
>> 
>> On 13.01.2019 13:14, Michael Tremer wrote:
>>> I sent squid thousands of requests with ab (Apache Benchmark) and that worked fine with only one redirector.
>> 
>> Thanks for the feedback! Merged. ;-)
>> 
>> 
> 
> Nevertheless it's worth reasoning about multiprocessing ( #redirectors = #processors ) versus multitasking ( #redirectors = #task needed by the environment ).
> Even with only one processor it is possible to serve multiple requests. Using wait phases for other processes is a basic feature of multitasking.

This is not true for the redirectors though.

Our version of squidGuard takes only one request at a time. While processing that, it performs a database lookup (which is a btree) which is held in memory. So at no time it is reading anything from disk which would allow the kernel to interrupt the process and wait until the hard drive has responded.

Therefore it does not make any sense to run more than one process per CPU core.

I guess update accelerator might suffer a little bit in the instance that it finds a match. Squidclamav is a different thing. It makes absolutely no sense to use that any more.

-Michael

> - Bernhard ( aka BeBiMa )

Re: Merge Window Closing Announcement for Core Update 127

[Bernhard Bitsch]

[-- Attachment #1: Type: text/plain, Size: 2132 bytes --]

Hi,

> Gesendet: Montag, 14. Januar 2019 um 11:43 Uhr
> Von: "Michael Tremer" <michael.tremer(a)ipfire.org>
> An: "Bernhard Bitsch" <Bernhard.Bitsch(a)gmx.de>
> Cc: "IPFire: Development-List" <development(a)lists.ipfire.org>
> Betreff: Re: Merge Window Closing Announcement for Core Update 127
>
> Hello,
> 
> > On 13 Jan 2019, at 14:24, Bernhard Bitsch <Bernhard.Bitsch(a)gmx.de> wrote:
> > 
> > 
> > 
> >> Gesendet: Sonntag, 13. Januar 2019 um 13:43 Uhr
> >> Von: "Matthias Fischer" <matthias.fischer(a)ipfire.org>
> >> An: "Michael Tremer" <michael.tremer(a)ipfire.org>
> >> Cc: "IPFire: Development-List" <development(a)lists.ipfire.org>
> >> Betreff: Re: Merge Window Closing Announcement for Core Update 127
> >> 
> >> Hi,
> >> 
> >> On 13.01.2019 13:14, Michael Tremer wrote:
> >>> I sent squid thousands of requests with ab (Apache Benchmark) and that worked fine with only one redirector.
> >> 
> >> Thanks for the feedback! Merged. ;-)
> >> 
> >> 
> > 
> > Nevertheless it's worth reasoning about multiprocessing ( #redirectors = #processors ) versus multitasking ( #redirectors = #task needed by the environment ).
> > Even with only one processor it is possible to serve multiple requests. Using wait phases for other processes is a basic feature of multitasking.
> 
> This is not true for the redirectors though.
> 
> Our version of squidGuard takes only one request at a time. While processing that, it performs a database lookup (which is a btree) which is held in memory. So at no time it is reading anything from disk which would allow the kernel to interrupt the process and wait until the hard drive has responded.
> 
> Therefore it does not make any sense to run more than one process per CPU core.
>

But multitasking isn't accomplished by I/O-waits only. Task switches occur through time slicing and signals also.
 
> I guess update accelerator might suffer a little bit in the instance that it finds a match. Squidclamav is a different thing. It makes absolutely no sense to use that any more.
> 
> -Michael
> 
> > - Bernhard ( aka BeBiMa )
> 
> 

- Bernhard

Re: Merge Window Closing Announcement for Core Update 127

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 2548 bytes --]


> On 14 Jan 2019, at 13:32, Bernhard Bitsch <Bernhard.Bitsch(a)gmx.de> wrote:
> 
> Hi,
> 
>> Gesendet: Montag, 14. Januar 2019 um 11:43 Uhr
>> Von: "Michael Tremer" <michael.tremer(a)ipfire.org>
>> An: "Bernhard Bitsch" <Bernhard.Bitsch(a)gmx.de>
>> Cc: "IPFire: Development-List" <development(a)lists.ipfire.org>
>> Betreff: Re: Merge Window Closing Announcement for Core Update 127
>> 
>> Hello,
>> 
>>> On 13 Jan 2019, at 14:24, Bernhard Bitsch <Bernhard.Bitsch(a)gmx.de> wrote:
>>> 
>>> 
>>> 
>>>> Gesendet: Sonntag, 13. Januar 2019 um 13:43 Uhr
>>>> Von: "Matthias Fischer" <matthias.fischer(a)ipfire.org>
>>>> An: "Michael Tremer" <michael.tremer(a)ipfire.org>
>>>> Cc: "IPFire: Development-List" <development(a)lists.ipfire.org>
>>>> Betreff: Re: Merge Window Closing Announcement for Core Update 127
>>>> 
>>>> Hi,
>>>> 
>>>> On 13.01.2019 13:14, Michael Tremer wrote:
>>>>> I sent squid thousands of requests with ab (Apache Benchmark) and that worked fine with only one redirector.
>>>> 
>>>> Thanks for the feedback! Merged. ;-)
>>>> 
>>>> 
>>> 
>>> Nevertheless it's worth reasoning about multiprocessing ( #redirectors = #processors ) versus multitasking ( #redirectors = #task needed by the environment ).
>>> Even with only one processor it is possible to serve multiple requests. Using wait phases for other processes is a basic feature of multitasking.
>> 
>> This is not true for the redirectors though.
>> 
>> Our version of squidGuard takes only one request at a time. While processing that, it performs a database lookup (which is a btree) which is held in memory. So at no time it is reading anything from disk which would allow the kernel to interrupt the process and wait until the hard drive has responded.
>> 
>> Therefore it does not make any sense to run more than one process per CPU core.
>> 
> 
> But multitasking isn't accomplished by I/O-waits only. Task switches occur through time slicing and signals also.

Yes, but that is not a case squidguard is likely to run into.

Processing a request is fast and won’t take minutes or even seconds.

If the process is being paused and something else is running then there is no point in having multiple processes waiting. One process waiting is enough.

>> I guess update accelerator might suffer a little bit in the instance that it finds a match. Squidclamav is a different thing. It makes absolutely no sense to use that any more.
>> 
>> -Michael
>> 
>>> - Bernhard ( aka BeBiMa )
>> 
>> 
> 
> - Bernhard