Hello Adolf, hello Michael,

thank you for discussing this and the insight.

Although this seems to have a different root cause, it reminds me of various update problems we've been trying to solve in https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=d0ba077ed3851346b1dd6d82... (which we never fully traced back, but ever since, no reports of such faulty behavior have surfaced again).

I was just wondering if it makes sense to run a "sync" before updating the linker cache as well. It wouldn't have fixed the Apache restart issue, but maybe makes things a little bit more robust otherwise?

Thanks, and best regards, Peter Müller

Good morning,

Yes, I can confirm this. This happened on one of the firewalls in our infrastructure.

It seems that the runtime linker cache isn’t up to date at the time Apache is being restarted, and therefore it simply fails to restart.

However, ldconfig is executed before restarting apache, so I am not sure why this is happening.

It is however safe to reboot the affected systems after this point and Apache will come back up just fine.

Best, -Michael

...

On 25 Nov 2023, at 09:25, Adolf Belka adolf.belka@ipfire.org wrote:

Hi All,

There has been a thread in the forum:-

https://community.ipfire.org/t/upgrade-to-181-hangs/10634/1

where four people have reported that apache has not restarted during the upgrade so their WUI hangs at the line:-

PAKFIRE UPGR: core-upgrade-181: Upgrading files and running post-upgrading scripts…

but their pakfire logs showed that the upgrade continued successfully. One person has done the upgrade on 14 systems. 3 went without any problem but 11 hung after the above line.

IPFire has continued working fine behind the scenes so no big interruption to their operation but clearly for some systems apache did not properly restart.

In future core updates, if apache is being upgraded and needs to be restarted, maybe it would be useful to add a check that apache did actually successfully restart and if not to try to start it again.

Any other thoughts on this.

Regards, Adolf.

-- Sent from my laptop

Hi Charles,

After the patch has been created, it then has to be merged into the Core-Update-181 branch. Then that merged branch has to be built via the Nightly Builds process and then that newly created iso has to replace the existing one in the downloads section.

The patch has been merged and the Nightly Build has been run successfully. That new iso is not yet available via the downloads link but you can download it now from the Nightly location if you don't want to wait for it to become available in the website downloads section.

https://nightly.ipfire.org/core181/2023-11-27%2011:26:00%20+0000-ece94c7e/x8...

I have presumed that you are using x86_64. If it is aarch64 then just click on Parent Directory to take you back up one level and then on the aarch64 directory.

If you check in the changelog.txt file you will see the commit for the apache change.

I am sure that the updated iso will be made available via the downloads website section before too long.

Regards, Adolf.

On 28/11/2023 14:27, Charles Brown wrote:

I have made a couple of attempts to the get the "fixed" update to Core 181.  Each time pakfire upgrade completes, I check the apache script in /etc/init.d and it is the same as before. That is, it does not reflect https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=ece94c7edfa45b724ef05b57... It still shows ...

stop)                 boot_mesg "Stopping Apache daemon..."                 killproc /usr/sbin/httpd                 evaluate_retval                 ;;

restart)                 boot_mesg "Restarting Apache daemon..."                 /usr/sbin/apachectl -k restart                 evaluate_retval                 ;;

I did indeed actually do the upgrade. Punched 'mine' back to 180 to make it do the upgrade thing Did a pakfire update --force, then did pakfire upgrade ...

CORE INFO: Checking for Core-Updates... CORE UPGR: Upgrading from release 180 to 181 core-upgrade-2.27... 100.00% |=============================>| 73.61 MB PAKFIRE UPGR: core-upgrade-181: Decrypting... PAKFIRE UPGR: core-upgrade-181: Upgrading files and running post-upgrading scripts... PAKFIRE UPGR: core-upgrade-181: Finished.

PAKFIRE INFO: Checking for package updates... PAKFIRE WARN: No new package upgrades available. .

Hi Charles,

You are correct. I replied on the basis of a fresh download but of course you are looking at the pakfire upgrade process.

My apologies for my incorrect feedback.

It is now down to the mirror updates and I don't know how long that should take and if all will update at the same time or if it is variable for each of the mirrors.

Regards, Adolf

On 28/11/2023 15:46, Charles Brown wrote:

I had assumed a pakfire update would do based on Michael's post at ... https://community.ipfire.org/t/upgrade-to-181-hangs/10634/30 "A fixed update has been pushed and as soon as all mirror servers have updated, this problem won’t arise any more."

Hi Michael,

There have been some people still talking about the WUI hanging after the update was done so I ran a vm check.

I cloned a CU180 system and then ran the update from pakfire. It came to the post-upgrade scripts and then it hung for a long time.

I checked via the console and the changes from your patch were in the initscript.

I then ran the status command

/etc/rc.d/init.d/apache status /usr/sbin/httpd is not running.

Then did a restart followed by another status

/etc/rc.d/init.d/apache start Starting Apache daemon...                  [  OK  ]

/etc/rc.d/init.d/apache status httpd is running with Process ID(s) 19645 19644 19643.

I then checked for messages in the Core Update 181 log and found the following

Stopping Apache daemon...                  [  OK  ] Starting Apache daemon... httpd (pid 6165) already running

So the new initscript was run but for some reason it thought that httpd was still running at that time.

So it looks like the initscript patch has been added in to the updater but is not fixing the problem of Apache not restarting properly.

Should I raise this as a bug or leave it with this email thread?

Regards,

Adolf.

On 28/11/2023 16:56, Michael Tremer wrote:

Hello Charles,

This is correct. The updater should ship the updated script - unless I made a mistake here.

I did not change the installation images, which will have the old script, but we will ship it with the next update again.

-Michael

...

On 28 Nov 2023, at 14:46, Charles Brown cab_77573@yahoo.com wrote:

I had assumed a pakfire update would do based on Michael's post at ... https://community.ipfire.org/t/upgrade-to-181-hangs/10634/30 "A fixed update has been pushed and as soon as all mirror servers have updated, this problem won’t arise any more."

Hi Arne,

I saw that you had done a mod to the update.sh script for the apache and udev restarts and that the nightly build for the CU181 had run.

So I did an update of a CU180 vm that I cloned and the WUI stayed accessible the whole time. So I can confirm that your fix has solved that problem.

Regards,

Adolf.

On 01/12/2023 13:38, Adolf Belka wrote:

Hi Michael,

Further update.

I re-ran a test on a vm but using the pakfire commands from the console (request from someone on the forum).

It failed again but with a different error message this time. I don't believe this was due to the different upgrade method but I suspect a timing related issue is more likely to be occurring.

The message in the log was:

|Stopping Apache daemon... [ OK ] Starting Apache daemon... (98)Address already in use: AH00072: make_sock: could not bind to address [::]:81 (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:81 no listening sockets available, shutting down|

Regards,

Adolf.

On 01/12/2023 13:13, Adolf Belka wrote:

...

Hi Michael,

There have been some people still talking about the WUI hanging after the update was done so I ran a vm check.

I cloned a CU180 system and then ran the update from pakfire. It came to the post-upgrade scripts and then it hung for a long time.

I checked via the console and the changes from your patch were in the initscript.

I then ran the status command

/etc/rc.d/init.d/apache status /usr/sbin/httpd is not running.

Then did a restart followed by another status

/etc/rc.d/init.d/apache start Starting Apache daemon...                  [  OK  ]

/etc/rc.d/init.d/apache status httpd is running with Process ID(s) 19645 19644 19643.

I then checked for messages in the Core Update 181 log and found the following

Stopping Apache daemon...                  [  OK  ] Starting Apache daemon... httpd (pid 6165) already running

So the new initscript was run but for some reason it thought that httpd was still running at that time.

So it looks like the initscript patch has been added in to the updater but is not fixing the problem of Apache not restarting properly.

Should I raise this as a bug or leave it with this email thread?

Regards,

Adolf.

On 28/11/2023 16:56, Michael Tremer wrote:

...

Hello Charles,

This is correct. The updater should ship the updated script - unless I made a mistake here.

I did not change the installation images, which will have the old script, but we will ship it with the next update again.

-Michael

...

On 28 Nov 2023, at 14:46, Charles Brown cab_77573@yahoo.com wrote:

I had assumed a pakfire update would do based on Michael's post at ... https://community.ipfire.org/t/upgrade-to-181-hangs/10634/30 "A fixed update has been pushed and as soon as all mirror servers have updated, this problem won’t arise any more."