Feedback on Core Update 155 Testing

Feedback on Core Update 155 Testing

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 1501 bytes --]

Dear all,

There is feedback on the forum about IPFire not rebooting after update 
and needing a soft reset. After that it boots again. I also found the 
same effect on my VM testbed system for testing.

https://community.ipfire.org/t/feedback-core-update-155-testing/4896

That symptom triggered my memory (a bit late) and I found the discussion 
about my sysvinit patch.

https://lists.ipfire.org/pipermail/development/2021-February/009403.html

So the fix never got created and it did not revert from the update.

Sorry for this mess. I will do a new patch to hopefully solve that 
problem for adding to Core Update 155, or you can revert sysvinit and I 
will make the update for the following Core Update.


I also found the same problem with ssh not working after the update as 
mentioned in the forum post.

Rebooting gave the following error message in the log.

/usr/sbin/sshd: /lib/libcrypt.so.1: version `XCRYPT_2.0' not found 
(required by /usr/sbin/sshd)                                    [FAIL]


People on the forum had deleted the /lib/libcrypt.so.1 link, which was 
made to /lib/libcrypt-2.32.so and remade it to /usr/lib/libcrypt.so.1 
and they have reported that ssh then works. I have not tried that fix yet.

I checked the OpenSSH update patch I created and I didn't find anything 
related to libcrypt or a change of libcrypt location from /lib to /usr/lib


Regards,

Adolf

-- 
Sent from my laptop

Re: Feedback on Core Update 155 Testing

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 1961 bytes --]

Hi Everyone,

On 17/03/2021 10:35, Adolf Belka wrote:
> Dear all,
>
> There is feedback on the forum about IPFire not rebooting after update 
> and needing a soft reset. After that it boots again. I also found the 
> same effect on my VM testbed system for testing.
>
> https://community.ipfire.org/t/feedback-core-update-155-testing/4896
>
> That symptom triggered my memory (a bit late) and I found the 
> discussion about my sysvinit patch.
>
> https://lists.ipfire.org/pipermail/development/2021-February/009403.html
>
> So the fix never got created and it did not revert from the update.
>
> Sorry for this mess. I will do a new patch to hopefully solve that 
> problem for adding to Core Update 155, or you can revert sysvinit and 
> I will make the update for the following Core Update.
>
Thinking about this while out walking the dog, I realised if the 
decision is to leave the updated package in Core Update 155 then I need 
to just create a patch for whatever fix is required to make it work 
properly on first boot. If the decision is to revert then I will need to 
create a patch for the full update again. I will work on creating a 
patch to just fix what is now in CU155 unless I hear differently from 
anyone.

Regards,

Adolf.

>
> I also found the same problem with ssh not working after the update as 
> mentioned in the forum post.
>
> Rebooting gave the following error message in the log.
>
> /usr/sbin/sshd: /lib/libcrypt.so.1: version `XCRYPT_2.0' not found 
> (required by /usr/sbin/sshd) [FAIL]
>
>
> People on the forum had deleted the /lib/libcrypt.so.1 link, which was 
> made to /lib/libcrypt-2.32.so and remade it to /usr/lib/libcrypt.so.1 
> and they have reported that ssh then works. I have not tried that fix 
> yet.
>
> I checked the OpenSSH update patch I created and I didn't find 
> anything related to libcrypt or a change of libcrypt location from 
> /lib to /usr/lib
>
>
> Regards,
>
> Adolf
>
-- 
Sent from my laptop

Re: Feedback on Core Update 155 Testing

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 3235 bytes --]

Hi All,

On 17/03/2021 11:51, Adolf Belka wrote:
> Hi Everyone,
>
> On 17/03/2021 10:35, Adolf Belka wrote:
>> Dear all,
>>
>> There is feedback on the forum about IPFire not rebooting after update and needing a soft reset. After that it boots again. I also found the same effect on my VM testbed system for testing.
>>
>> https://community.ipfire.org/t/feedback-core-update-155-testing/4896
>>
>> That symptom triggered my memory (a bit late) and I found the discussion about my sysvinit patch.
>>
>> https://lists.ipfire.org/pipermail/development/2021-February/009403.html
>>
>> So the fix never got created and it did not revert from the update.
>>
>> Sorry for this mess. I will do a new patch to hopefully solve that problem for adding to Core Update 155, or you can revert sysvinit and I will make the update for the following Core Update.
>>
> Thinking about this while out walking the dog, I realised if the decision is to leave the updated package in Core Update 155 then I need to just create a patch for whatever fix is required to make it work properly on first boot. If the decision is to revert then I will need to create a patch for the full update again. I will work on creating a patch to just fix what is now in CU155 unless I hear differently from anyone.
>
> Regards,
>
> Adolf.
>
Back in early 2018 the sysvinit team decided to move the initctl pipe from /dev to /run.

The commit message for this was "FreeBSD apparently does not like named pipes in /dev, so we move it to /run for better cross-platform compatibility."


I installed CU155 from scratch and I had no problem at all. The pipe was in /run and nothing in /dev.

It is when an upgrade is carried out and the system starts with the pipe in /dev, then the upgrade is not changing this. It is only changed after a forced reboot by doing a soft reset.

It seems that we need some lines to remove the pipe from /dev and create it in /run at an appropriate stage of the upgrade. Is the lfs file the correct place to have these commands after the install commands or should they be in the file used to carry out the Core Update. The lfs file doesn't seem the correct place for this as it is only needed once for the upgrade from CU154 to CU155. After that the pipe will always stay in /run.


I would appreciate advice on the best approach. If it should be in the upgrade script, where is that located? If it should be in the lfs do I just put the mknod command after the install command.


Regards,

Adolf.

>>
>> I also found the same problem with ssh not working after the update as mentioned in the forum post.
>>
>> Rebooting gave the following error message in the log.
>>
>> /usr/sbin/sshd: /lib/libcrypt.so.1: version `XCRYPT_2.0' not found (required by /usr/sbin/sshd) [FAIL]
>>
>>
>> People on the forum had deleted the /lib/libcrypt.so.1 link, which was made to /lib/libcrypt-2.32.so and remade it to /usr/lib/libcrypt.so.1 and they have reported that ssh then works. I have not tried that fix yet.
>>
>> I checked the OpenSSH update patch I created and I didn't find anything related to libcrypt or a change of libcrypt location from /lib to /usr/lib
>>
>>
>> Regards,
>>
>> Adolf
>>

Re: Feedback on Core Update 155 Testing

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 2172 bytes --]

Hello,

> On 17 Mar 2021, at 09:35, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Dear all,
> 
> There is feedback on the forum about IPFire not rebooting after update and needing a soft reset. After that it boots again. I also found the same effect on my VM testbed system for testing.
> 
> https://community.ipfire.org/t/feedback-core-update-155-testing/4896
> 
> That symptom triggered my memory (a bit late) and I found the discussion about my sysvinit patch.
> 
> https://lists.ipfire.org/pipermail/development/2021-February/009403.html
> 
> So the fix never got created and it did not revert from the update.
> 
> Sorry for this mess. I will do a new patch to hopefully solve that problem for adding to Core Update 155, or you can revert sysvinit and I will make the update for the following Core Update.

No worries. I would only blame myself for forgetting to find a solution for this.

Arne simply created a symlink which temporarily makes things work again until the next reboot. After that, things are fine.

> I also found the same problem with ssh not working after the update as mentioned in the forum post.
> 
> Rebooting gave the following error message in the log.
> 
> /usr/sbin/sshd: /lib/libcrypt.so.1: version `XCRYPT_2.0' not found (required by /usr/sbin/sshd)                                    [FAIL]

This is another one that I fucked up myself :)

I do not know why though, because I tested an earlier version of the update on my machine and it worked fine.

> People on the forum had deleted the /lib/libcrypt.so.1 link, which was made to /lib/libcrypt-2.32.so and remade it to /usr/lib/libcrypt.so.1 and they have reported that ssh then works. I have not tried that fix yet.

I suppose Arne tested it and committed it.

> I checked the OpenSSH update patch I created and I didn't find anything related to libcrypt or a change of libcrypt location from /lib to /usr/lib

This was part of glibc deprecating libcrypt. Therefore I added a fork which is actively maintained.

Thanks for looking into these problems.

-Michael

> 
> 
> Regards,
> 
> Adolf
> 
> -- 
> Sent from my laptop
> 

Re: Feedback on Core Update 155 Testing

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 3654 bytes --]

Hello,

> On 17 Mar 2021, at 12:40, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi All,
> 
> On 17/03/2021 11:51, Adolf Belka wrote:
>> Hi Everyone,
>> 
>> On 17/03/2021 10:35, Adolf Belka wrote:
>>> Dear all,
>>> 
>>> There is feedback on the forum about IPFire not rebooting after update and needing a soft reset. After that it boots again. I also found the same effect on my VM testbed system for testing.
>>> 
>>> https://community.ipfire.org/t/feedback-core-update-155-testing/4896
>>> 
>>> That symptom triggered my memory (a bit late) and I found the discussion about my sysvinit patch.
>>> 
>>> https://lists.ipfire.org/pipermail/development/2021-February/009403.html
>>> 
>>> So the fix never got created and it did not revert from the update.
>>> 
>>> Sorry for this mess. I will do a new patch to hopefully solve that problem for adding to Core Update 155, or you can revert sysvinit and I will make the update for the following Core Update.
>>> 
>> Thinking about this while out walking the dog, I realised if the decision is to leave the updated package in Core Update 155 then I need to just create a patch for whatever fix is required to make it work properly on first boot. If the decision is to revert then I will need to create a patch for the full update again. I will work on creating a patch to just fix what is now in CU155 unless I hear differently from anyone.
>> 
>> Regards,
>> 
>> Adolf.
>> 
> Back in early 2018 the sysvinit team decided to move the initctl pipe from /dev to /run.
> 
> The commit message for this was "FreeBSD apparently does not like named pipes in /dev, so we move it to /run for better cross-platform compatibility."
> 
> 
> I installed CU155 from scratch and I had no problem at all. The pipe was in /run and nothing in /dev.
> 
> It is when an upgrade is carried out and the system starts with the pipe in /dev, then the upgrade is not changing this. It is only changed after a forced reboot by doing a soft reset.
> 
> It seems that we need some lines to remove the pipe from /dev and create it in /run at an appropriate stage of the upgrade. Is the lfs file the correct place to have these commands after the install commands or should they be in the file used to carry out the Core Update. The lfs file doesn't seem the correct place for this as it is only needed once for the upgrade from CU154 to CU155. After that the pipe will always stay in /run.
> 
> 
> I would appreciate advice on the best approach. If it should be in the upgrade script, where is that located? If it should be in the lfs do I just put the mknod command after the install command.

Arne applied this which should be good enough until the system reboots and creates the new pipe in the right place:

https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=5d747ddb70e74c9f5503618cc8aaa79b8f728cb3

-Michael

> 
> Regards,
> 
> Adolf.
> 
>>> 
>>> I also found the same problem with ssh not working after the update as mentioned in the forum post.
>>> 
>>> Rebooting gave the following error message in the log.
>>> 
>>> /usr/sbin/sshd: /lib/libcrypt.so.1: version `XCRYPT_2.0' not found (required by /usr/sbin/sshd) [FAIL]
>>> 
>>> 
>>> People on the forum had deleted the /lib/libcrypt.so.1 link, which was made to /lib/libcrypt-2.32.so and remade it to /usr/lib/libcrypt.so.1 and they have reported that ssh then works. I have not tried that fix yet.
>>> 
>>> I checked the OpenSSH update patch I created and I didn't find anything related to libcrypt or a change of libcrypt location from /lib to /usr/lib
>>> 
>>> 
>>> Regards,
>>> 
>>> Adolf

Re: Feedback on Core Update 155 Testing

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 5650 bytes --]

Hi All,

Reran the CU155 upgrade on my testbed VM. The fixes from Arne worked perfectly. No problems rebooting after the upgrade and was able to ssh in to the system.

I had every service turned on except IPSec VPN. After the upgrade, everything was running as before. Same after the reboot.

wio and bacula addons are working as expected after the upgrade.

I selected all graphs with hourly resolution and all was working as I would expect it to.

All ALG options gone on FW options WUI page.


Everything worked as expected, with no problems so far. I expect it to stay like that now.

I will try it over the next few days and confirm that everything stays stable.

Regards,

Adolf.

On 18/03/2021 12:31, Michael Tremer wrote:
> *thumbs up*
> 
>> On 17 Mar 2021, at 17:11, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi Jonatan & All,
>>
>> Thanks for the feedback. Glad it is all solved. I would rather do some investigation that ends up not being needed than not doing something that then should have been done.
>>
>> I will clone a new CU154 VM and test the update again tomorrow.
>>
>> Regards,
>>
>> Adolf.
>>
>> On 17/03/2021 16:22, Jonatan Schlag wrote:
>>> Sorry for missing the list :-(.
>>> ————————-
>>>
>>> Hi Adolf,
>>>
>>> thanks for having an eye on the community and reporting things back.
>>> Arne has already fixed that (see https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=5d747ddb70e74c9f5503618cc8aaa79b8f728cb3 <https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=5d747ddb70e74c9f5503618cc8aaa79b8f728cb3> and other comments in the current master).
>>>
>>> Sorry for answering this so late. There was communication on this over different places (community, chat, here, ...) and so things get lost somehow. I do not know if this could be improved, but at least I want to say sorry that you were not aware that the problem was already fixed.
>>>
>>> Greetings Jonatan
>>>
>>>>> Am 17.03.2021 um 13:40 schrieb Adolf Belka <adolf.belka(a)ipfire.org>:
>>>>>
>>>>> Hi All,
>>>>>
>>>>> On 17/03/2021 11:51, Adolf Belka wrote:
>>>>>> Hi Everyone,
>>>>>>
>>>>>> On 17/03/2021 10:35, Adolf Belka wrote:
>>>>>>> Dear all,
>>>>>>>
>>>>>>> There is feedback on the forum about IPFire not rebooting after update and needing a soft reset. After that it boots again. I also found the same effect on my VM testbed system for testing.
>>>>>>>
>>>>>>> https://community.ipfire.org/t/feedback-core-update-155-testing/4896
>>>>>>>
>>>>>>> That symptom triggered my memory (a bit late) and I found the discussion about my sysvinit patch.
>>>>>>>
>>>>>>> https://lists.ipfire.org/pipermail/development/2021-February/009403.html
>>>>>>>
>>>>>>> So the fix never got created and it did not revert from the update.
>>>>>>>
>>>>>>> Sorry for this mess. I will do a new patch to hopefully solve that problem for adding to Core Update 155, or you can revert sysvinit and I will make the update for the following Core Update.
>>>>>>>
>>>>>> Thinking about this while out walking the dog, I realised if the decision is to leave the updated package in Core Update 155 then I need to just create a patch for whatever fix is required to make it work properly on first boot. If the decision is to revert then I will need to create a patch for the full update again. I will work on creating a patch to just fix what is now in CU155 unless I hear differently from anyone.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Adolf.
>>>>>>
>>>>> Back in early 2018 the sysvinit team decided to move the initctl pipe from /dev to /run.
>>>>>
>>>>> The commit message for this was "FreeBSD apparently does not like named pipes in /dev, so we move it to /run for better cross-platform compatibility."
>>>>>
>>>>>
>>>>> I installed CU155 from scratch and I had no problem at all. The pipe was in /run and nothing in /dev.
>>>>>
>>>>> It is when an upgrade is carried out and the system starts with the pipe in /dev, then the upgrade is not changing this. It is only changed after a forced reboot by doing a soft reset.
>>>>>
>>>>> It seems that we need some lines to remove the pipe from /dev and create it in /run at an appropriate stage of the upgrade. Is the lfs file the correct place to have these commands after the install commands or should they be in the file used to carry out the Core Update. The lfs file doesn't seem the correct place for this as it is only needed once for the upgrade from CU154 to CU155. After that the pipe will always stay in /run.
>>>>>
>>>>>
>>>>> I would appreciate advice on the best approach. If it should be in the upgrade script, where is that located? If it should be in the lfs do I just put the mknod command after the install command.
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Adolf.
>>>>>
>>>>>>>
>>>>>>> I also found the same problem with ssh not working after the update as mentioned in the forum post.
>>>>>>>
>>>>>>> Rebooting gave the following error message in the log.
>>>>>>>
>>>>>>> /usr/sbin/sshd: /lib/libcrypt.so.1: version `XCRYPT_2.0' not found (required by /usr/sbin/sshd) [FAIL]
>>>>>>>
>>>>>>>
>>>>>>> People on the forum had deleted the /lib/libcrypt.so.1 link, which was made to /lib/libcrypt-2.32.so and remade it to /usr/lib/libcrypt.so.1 and they have reported that ssh then works. I have not tried that fix yet.
>>>>>>>
>>>>>>> I checked the OpenSSH update patch I created and I didn't find anything related to libcrypt or a change of libcrypt location from /lib to /usr/lib
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Adolf
>>>>>>>
>> -- 
>> Sent from my laptop
> 

Re: Feedback on Core Update 155 Testing

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 4846 bytes --]

*thumbs up*

> On 17 Mar 2021, at 17:11, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Jonatan & All,
> 
> Thanks for the feedback. Glad it is all solved. I would rather do some investigation that ends up not being needed than not doing something that then should have been done.
> 
> I will clone a new CU154 VM and test the update again tomorrow.
> 
> Regards,
> 
> Adolf.
> 
> On 17/03/2021 16:22, Jonatan Schlag wrote:
>> Sorry for missing the list :-(.
>> ————————-
>> 
>> Hi Adolf,
>> 
>> thanks for having an eye on the community and reporting things back.
>> Arne has already fixed that (see https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=5d747ddb70e74c9f5503618cc8aaa79b8f728cb3 <https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=5d747ddb70e74c9f5503618cc8aaa79b8f728cb3> and other comments in the current master).
>> 
>> Sorry for answering this so late. There was communication on this over different places (community, chat, here, ...) and so things get lost somehow. I do not know if this could be improved, but at least I want to say sorry that you were not aware that the problem was already fixed.
>> 
>> Greetings Jonatan
>> 
>>>> Am 17.03.2021 um 13:40 schrieb Adolf Belka <adolf.belka(a)ipfire.org>:
>>>> 
>>>> Hi All,
>>>> 
>>>> On 17/03/2021 11:51, Adolf Belka wrote:
>>>>> Hi Everyone,
>>>>> 
>>>>> On 17/03/2021 10:35, Adolf Belka wrote:
>>>>>> Dear all,
>>>>>> 
>>>>>> There is feedback on the forum about IPFire not rebooting after update and needing a soft reset. After that it boots again. I also found the same effect on my VM testbed system for testing.
>>>>>> 
>>>>>> https://community.ipfire.org/t/feedback-core-update-155-testing/4896
>>>>>> 
>>>>>> That symptom triggered my memory (a bit late) and I found the discussion about my sysvinit patch.
>>>>>> 
>>>>>> https://lists.ipfire.org/pipermail/development/2021-February/009403.html
>>>>>> 
>>>>>> So the fix never got created and it did not revert from the update.
>>>>>> 
>>>>>> Sorry for this mess. I will do a new patch to hopefully solve that problem for adding to Core Update 155, or you can revert sysvinit and I will make the update for the following Core Update.
>>>>>> 
>>>>> Thinking about this while out walking the dog, I realised if the decision is to leave the updated package in Core Update 155 then I need to just create a patch for whatever fix is required to make it work properly on first boot. If the decision is to revert then I will need to create a patch for the full update again. I will work on creating a patch to just fix what is now in CU155 unless I hear differently from anyone.
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> Adolf.
>>>>> 
>>>> Back in early 2018 the sysvinit team decided to move the initctl pipe from /dev to /run.
>>>> 
>>>> The commit message for this was "FreeBSD apparently does not like named pipes in /dev, so we move it to /run for better cross-platform compatibility."
>>>> 
>>>> 
>>>> I installed CU155 from scratch and I had no problem at all. The pipe was in /run and nothing in /dev.
>>>> 
>>>> It is when an upgrade is carried out and the system starts with the pipe in /dev, then the upgrade is not changing this. It is only changed after a forced reboot by doing a soft reset.
>>>> 
>>>> It seems that we need some lines to remove the pipe from /dev and create it in /run at an appropriate stage of the upgrade. Is the lfs file the correct place to have these commands after the install commands or should they be in the file used to carry out the Core Update. The lfs file doesn't seem the correct place for this as it is only needed once for the upgrade from CU154 to CU155. After that the pipe will always stay in /run.
>>>> 
>>>> 
>>>> I would appreciate advice on the best approach. If it should be in the upgrade script, where is that located? If it should be in the lfs do I just put the mknod command after the install command.
>>>> 
>>>> 
>>>> Regards,
>>>> 
>>>> Adolf.
>>>> 
>>>>>> 
>>>>>> I also found the same problem with ssh not working after the update as mentioned in the forum post.
>>>>>> 
>>>>>> Rebooting gave the following error message in the log.
>>>>>> 
>>>>>> /usr/sbin/sshd: /lib/libcrypt.so.1: version `XCRYPT_2.0' not found (required by /usr/sbin/sshd) [FAIL]
>>>>>> 
>>>>>> 
>>>>>> People on the forum had deleted the /lib/libcrypt.so.1 link, which was made to /lib/libcrypt-2.32.so and remade it to /usr/lib/libcrypt.so.1 and they have reported that ssh then works. I have not tried that fix yet.
>>>>>> 
>>>>>> I checked the OpenSSH update patch I created and I didn't find anything related to libcrypt or a change of libcrypt location from /lib to /usr/lib
>>>>>> 
>>>>>> 
>>>>>> Regards,
>>>>>> 
>>>>>> Adolf
>>>>>> 
> -- 
> Sent from my laptop