Apache not properly restarting for some people with Core Update 181 Release

Apache not properly restarting for some people with Core Update 181 Release

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 931 bytes --]

Hi All,

There has been a thread in the forum:-

https://community.ipfire.org/t/upgrade-to-181-hangs/10634/1

where four people have reported that apache has not restarted during the 
upgrade so their WUI hangs at the line:-

PAKFIRE UPGR: core-upgrade-181: Upgrading files and running 
post-upgrading scripts…

but their pakfire logs showed that the upgrade continued successfully. 
One person has done the upgrade on 14 systems. 3 went without any 
problem but 11 hung after the above line.

IPFire has continued working fine behind the scenes so no big 
interruption to their operation but clearly for some systems apache did 
not properly restart.

In future core updates, if apache is being upgraded and needs to be 
restarted, maybe it would be useful to add a check that apache did 
actually successfully restart and if not to try to start it again.

Any other thoughts on this.

Regards,
Adolf.

-- 
Sent from my laptop

Re: Apache not properly restarting for some people with Core Update 181 Release

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1552 bytes --]

Good morning,

Yes, I can confirm this. This happened on one of the firewalls in our infrastructure.

It seems that the runtime linker cache isn’t up to date at the time Apache is being restarted, and therefore it simply fails to restart.

However, ldconfig is executed before restarting apache, so I am not sure why this is happening.

It is however safe to reboot the affected systems after this point and Apache will come back up just fine.

Best,
-Michael

> On 25 Nov 2023, at 09:25, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi All,
> 
> There has been a thread in the forum:-
> 
> https://community.ipfire.org/t/upgrade-to-181-hangs/10634/1
> 
> where four people have reported that apache has not restarted during the upgrade so their WUI hangs at the line:-
> 
> PAKFIRE UPGR: core-upgrade-181: Upgrading files and running post-upgrading scripts…
> 
> but their pakfire logs showed that the upgrade continued successfully. One person has done the upgrade on 14 systems. 3 went without any problem but 11 hung after the above line.
> 
> IPFire has continued working fine behind the scenes so no big interruption to their operation but clearly for some systems apache did not properly restart.
> 
> In future core updates, if apache is being upgraded and needs to be restarted, maybe it would be useful to add a check that apache did actually successfully restart and if not to try to start it again.
> 
> Any other thoughts on this.
> 
> Regards,
> Adolf.
> 
> -- 
> Sent from my laptop
> 

Re: Apache not properly restarting for some people with Core Update 181 Release

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 2262 bytes --]

Hello Adolf, hello Michael,

thank you for discussing this and the insight.

Although this seems to have a different root cause, it reminds me
of various update problems we've been trying to solve in
https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=d0ba077ed3851346b1dd6d82867103df8446aea8
(which we never fully traced back, but ever since, no reports of
such faulty behavior have surfaced again).

I was just wondering if it makes sense to run a "sync" before updating
the linker cache as well. It wouldn't have fixed the Apache restart
issue, but maybe makes things a little bit more robust otherwise?

Thanks, and best regards,
Peter Müller


> Good morning,
> 
> Yes, I can confirm this. This happened on one of the firewalls in our infrastructure.
> 
> It seems that the runtime linker cache isn’t up to date at the time Apache is being restarted, and therefore it simply fails to restart.
> 
> However, ldconfig is executed before restarting apache, so I am not sure why this is happening.
> 
> It is however safe to reboot the affected systems after this point and Apache will come back up just fine.
> 
> Best,
> -Michael
> 
>> On 25 Nov 2023, at 09:25, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi All,
>>
>> There has been a thread in the forum:-
>>
>> https://community.ipfire.org/t/upgrade-to-181-hangs/10634/1
>>
>> where four people have reported that apache has not restarted during the upgrade so their WUI hangs at the line:-
>>
>> PAKFIRE UPGR: core-upgrade-181: Upgrading files and running post-upgrading scripts…
>>
>> but their pakfire logs showed that the upgrade continued successfully. One person has done the upgrade on 14 systems. 3 went without any problem but 11 hung after the above line.
>>
>> IPFire has continued working fine behind the scenes so no big interruption to their operation but clearly for some systems apache did not properly restart.
>>
>> In future core updates, if apache is being upgraded and needs to be restarted, maybe it would be useful to add a check that apache did actually successfully restart and if not to try to start it again.
>>
>> Any other thoughts on this.
>>
>> Regards,
>> Adolf.
>>
>> -- 
>> Sent from my laptop
>>
> 

Re: Apache not properly restarting for some people with Core Update 181 Release

[Charles Brown]

[-- Attachment #1: Type: text/plain, Size: 1395 bytes --]

I have made a couple of attempts to the get the "fixed" update to Core 
181.  Each time pakfire upgrade completes, I check the apache script in 
/etc/init.d and it is the same as before.
That is, it does not reflect 
https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=ece94c7edfa45b724ef05b57ff7a8a18c43fb99b
It still shows ...

         stop)
                 boot_mesg "Stopping Apache daemon..."
                 killproc /usr/sbin/httpd
                 evaluate_retval
                 ;;

         restart)
                 boot_mesg "Restarting Apache daemon..."
                 /usr/sbin/apachectl -k restart
                 evaluate_retval
                 ;;

I did indeed actually do the upgrade.
Punched 'mine' back to 180 to make it do the upgrade thing
Did a pakfire update --force, then did pakfire upgrade ...

CORE INFO: Checking for Core-Updates...
CORE UPGR: Upgrading from release 180 to 181
core-upgrade-2.27... 100.00% |=============================>| 73.61 MB
PAKFIRE UPGR: core-upgrade-181: Decrypting...
PAKFIRE UPGR: core-upgrade-181: Upgrading files and running 
post-upgrading scripts...
PAKFIRE UPGR: core-upgrade-181: Finished.

PAKFIRE INFO: Checking for package updates...
PAKFIRE WARN: No new package upgrades available.
.

Re: Apache not properly restarting for some people with Core Update 181 Release

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 2592 bytes --]

Hi Charles,

After the patch has been created, it then has to be merged into the 
Core-Update-181 branch. Then that merged branch has to be built via the 
Nightly Builds process and then that newly created iso has to replace 
the existing one in the downloads section.

The patch has been merged and the Nightly Build has been run 
successfully. That new iso is not yet available via the downloads link 
but you can download it now from the Nightly location if you don't want 
to wait for it to become available in the website downloads section.

https://nightly.ipfire.org/core181/2023-11-27%2011:26:00%20+0000-ece94c7e/x86_64/

I have presumed that you are using x86_64. If it is aarch64 then just 
click on Parent Directory to take you back up one level and then on the 
aarch64 directory.

If you check in the changelog.txt file you will see the commit for the 
apache change.

I am sure that the updated iso will be made available via the downloads 
website section before too long.

Regards,
Adolf.


On 28/11/2023 14:27, Charles Brown wrote:
> I have made a couple of attempts to the get the "fixed" update to Core 
> 181.  Each time pakfire upgrade completes, I check the apache script 
> in /etc/init.d and it is the same as before.
> That is, it does not reflect 
> https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=ece94c7edfa45b724ef05b57ff7a8a18c43fb99b
> It still shows ...
>
>         stop)
>                 boot_mesg "Stopping Apache daemon..."
>                 killproc /usr/sbin/httpd
>                 evaluate_retval
>                 ;;
>
>         restart)
>                 boot_mesg "Restarting Apache daemon..."
>                 /usr/sbin/apachectl -k restart
>                 evaluate_retval
>                 ;;
>
> I did indeed actually do the upgrade.
> Punched 'mine' back to 180 to make it do the upgrade thing
> Did a pakfire update --force, then did pakfire upgrade ...
>
> CORE INFO: Checking for Core-Updates...
> CORE UPGR: Upgrading from release 180 to 181
> core-upgrade-2.27... 100.00% |=============================>| 73.61 MB
> PAKFIRE UPGR: core-upgrade-181: Decrypting...
> PAKFIRE UPGR: core-upgrade-181: Upgrading files and running 
> post-upgrading scripts...
> PAKFIRE UPGR: core-upgrade-181: Finished.
>
> PAKFIRE INFO: Checking for package updates...
> PAKFIRE WARN: No new package upgrades available.
> .
>

-- 
Sent from my laptop

Re: Apache not properly restarting for some people with Core Update 181 Release

[Charles Brown]

[-- Attachment #1: Type: text/plain, Size: 252 bytes --]

I had assumed a pakfire update would do based on Michael's post at ...
https://community.ipfire.org/t/upgrade-to-181-hangs/10634/30
"A fixed update has been pushed and as soon as all mirror servers have 
updated, this problem won’t arise any more."

Re: Apache not properly restarting for some people with Core Update 181 Release

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 692 bytes --]

Hi Charles,

You are correct. I replied on the basis of a fresh download but of 
course you are looking at the pakfire upgrade process.

My apologies for my incorrect feedback.

It is now down to the mirror updates and I don't know how long that 
should take and if all will update at the same time or if it is variable 
for each of the mirrors.

Regards,
Adolf

On 28/11/2023 15:46, Charles Brown wrote:
> I had assumed a pakfire update would do based on Michael's post at ...
> https://community.ipfire.org/t/upgrade-to-181-hangs/10634/30
> "A fixed update has been pushed and as soon as all mirror servers have 
> updated, this problem won’t arise any more."
>

-- 
Sent from my laptop

Re: Apache not properly restarting for some people with Core Update 181 Release

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 593 bytes --]

Hello Charles,

This is correct. The updater should ship the updated script - unless I made a mistake here.

I did not change the installation images, which will have the old script, but we will ship it with the next update again.

-Michael

> On 28 Nov 2023, at 14:46, Charles Brown <cab_77573(a)yahoo.com> wrote:
> 
> I had assumed a pakfire update would do based on Michael's post at ...
> https://community.ipfire.org/t/upgrade-to-181-hangs/10634/30
> "A fixed update has been pushed and as soon as all mirror servers have updated, this problem won’t arise any more."
> 

Re: Apache not properly restarting for some people with Core Update 181 Release

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 1903 bytes --]

Hi Michael,

There have been some people still talking about the WUI hanging after the update was done so I ran a vm check.


I cloned a CU180 system and then ran the update from pakfire. It came to the post-upgrade scripts and then it hung for a long time.

I checked via the console and the changes from your patch were in the initscript.

I then ran the status command

/etc/rc.d/init.d/apache status
/usr/sbin/httpd is not running.

Then did a restart followed by another status

/etc/rc.d/init.d/apache start
Starting Apache daemon...                  [  OK  ]

/etc/rc.d/init.d/apache status
httpd is running with Process ID(s) 19645 19644 19643.


I then checked for messages in the Core Update 181 log and found the following

Stopping Apache daemon...                  [  OK  ]
Starting Apache daemon...
httpd (pid 6165) already running

So the new initscript was run but for some reason it thought that httpd was still running at that time.

So it looks like the initscript patch has been added in to the updater but is not fixing the problem of Apache not restarting properly.

Should I raise this as a bug or leave it with this email thread?

Regards,

Adolf.


On 28/11/2023 16:56, Michael Tremer wrote:
> Hello Charles,
>
> This is correct. The updater should ship the updated script - unless I made a mistake here.
>
> I did not change the installation images, which will have the old script, but we will ship it with the next update again.
>
> -Michael
>
>> On 28 Nov 2023, at 14:46, Charles Brown <cab_77573(a)yahoo.com> wrote:
>>
>> I had assumed a pakfire update would do based on Michael's post at ...
>> https://community.ipfire.org/t/upgrade-to-181-hangs/10634/30
>> "A fixed update has been pushed and as soon as all mirror servers have updated, this problem won’t arise any more."
>>

Re: Apache not properly restarting for some people with Core Update 181 Release

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 2694 bytes --]

Hi Michael,

Further update.

I re-ran a test on a vm but using the pakfire commands from the console (request from someone on the forum).

It failed again but with a different error message this time. I don't believe this was due to the different upgrade method but I suspect a timing related issue is more likely to be occurring.

The message in the log was:

|Stopping Apache daemon... [ OK ] Starting Apache daemon... (98)Address already in use: AH00072: make_sock: could not bind to address [::]:81 (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:81 no listening sockets available, shutting down|

Regards,

Adolf.


On 01/12/2023 13:13, Adolf Belka wrote:
> Hi Michael,
>
> There have been some people still talking about the WUI hanging after the update was done so I ran a vm check.
>
>
> I cloned a CU180 system and then ran the update from pakfire. It came to the post-upgrade scripts and then it hung for a long time.
>
> I checked via the console and the changes from your patch were in the initscript.
>
> I then ran the status command
>
> /etc/rc.d/init.d/apache status
> /usr/sbin/httpd is not running.
>
> Then did a restart followed by another status
>
> /etc/rc.d/init.d/apache start
> Starting Apache daemon...                  [  OK  ]
>
> /etc/rc.d/init.d/apache status
> httpd is running with Process ID(s) 19645 19644 19643.
>
>
> I then checked for messages in the Core Update 181 log and found the following
>
> Stopping Apache daemon...                  [  OK  ]
> Starting Apache daemon...
> httpd (pid 6165) already running
>
> So the new initscript was run but for some reason it thought that httpd was still running at that time.
>
> So it looks like the initscript patch has been added in to the updater but is not fixing the problem of Apache not restarting properly.
>
> Should I raise this as a bug or leave it with this email thread?
>
> Regards,
>
> Adolf.
>
>
> On 28/11/2023 16:56, Michael Tremer wrote:
>> Hello Charles,
>>
>> This is correct. The updater should ship the updated script - unless I made a mistake here.
>>
>> I did not change the installation images, which will have the old script, but we will ship it with the next update again.
>>
>> -Michael
>>
>>> On 28 Nov 2023, at 14:46, Charles Brown <cab_77573(a)yahoo.com> wrote:
>>>
>>> I had assumed a pakfire update would do based on Michael's post at ...
>>> https://community.ipfire.org/t/upgrade-to-181-hangs/10634/30
>>> "A fixed update has been pushed and as soon as all mirror servers have updated, this problem won’t arise any more."
>>>

Re: Apache not properly restarting for some people with Core Update 181 Release

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 3145 bytes --]

Hi Arne,

I saw that you had done a mod to the update.sh script for the apache and udev restarts and that the nightly build for the CU181 had run.

So I did an update of a CU180 vm that I cloned and the WUI stayed accessible the whole time. So I can confirm that your fix has solved that problem.

Regards,

Adolf.


On 01/12/2023 13:38, Adolf Belka wrote:
> Hi Michael,
>
> Further update.
>
> I re-ran a test on a vm but using the pakfire commands from the console (request from someone on the forum).
>
> It failed again but with a different error message this time. I don't believe this was due to the different upgrade method but I suspect a timing related issue is more likely to be occurring.
>
> The message in the log was:
>
> |Stopping Apache daemon... [ OK ] Starting Apache daemon... (98)Address already in use: AH00072: make_sock: could not bind to address [::]:81 (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:81 no listening sockets available, shutting down|
>
> Regards,
>
> Adolf.
>
>
> On 01/12/2023 13:13, Adolf Belka wrote:
>> Hi Michael,
>>
>> There have been some people still talking about the WUI hanging after the update was done so I ran a vm check.
>>
>>
>> I cloned a CU180 system and then ran the update from pakfire. It came to the post-upgrade scripts and then it hung for a long time.
>>
>> I checked via the console and the changes from your patch were in the initscript.
>>
>> I then ran the status command
>>
>> /etc/rc.d/init.d/apache status
>> /usr/sbin/httpd is not running.
>>
>> Then did a restart followed by another status
>>
>> /etc/rc.d/init.d/apache start
>> Starting Apache daemon...                  [  OK  ]
>>
>> /etc/rc.d/init.d/apache status
>> httpd is running with Process ID(s) 19645 19644 19643.
>>
>>
>> I then checked for messages in the Core Update 181 log and found the following
>>
>> Stopping Apache daemon...                  [  OK  ]
>> Starting Apache daemon...
>> httpd (pid 6165) already running
>>
>> So the new initscript was run but for some reason it thought that httpd was still running at that time.
>>
>> So it looks like the initscript patch has been added in to the updater but is not fixing the problem of Apache not restarting properly.
>>
>> Should I raise this as a bug or leave it with this email thread?
>>
>> Regards,
>>
>> Adolf.
>>
>>
>> On 28/11/2023 16:56, Michael Tremer wrote:
>>> Hello Charles,
>>>
>>> This is correct. The updater should ship the updated script - unless I made a mistake here.
>>>
>>> I did not change the installation images, which will have the old script, but we will ship it with the next update again.
>>>
>>> -Michael
>>>
>>>> On 28 Nov 2023, at 14:46, Charles Brown <cab_77573(a)yahoo.com> wrote:
>>>>
>>>> I had assumed a pakfire update would do based on Michael's post at ...
>>>> https://community.ipfire.org/t/upgrade-to-181-hangs/10634/30
>>>> "A fixed update has been pushed and as soon as all mirror servers have updated, this problem won’t arise any more."
>>>>