Hi All,
I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
Normally my updates go without any real problems. This is the first time where this is not the case.
First try:-
After running the update, which went quite quickly, the bottom of the screen had
Core Update 167 Development Build: master/c22d834c
I couldn't remember if this is what is expected or not for Testing releases.
Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
Lots of ipset restore error but went by too fast to get more detail.
Tried to access via the WUI and no access. Tried via SSH and no access.
Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
starting DUID 00:04:70:........ red0: interface not found
So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
Looked in bootlog and this was filled with around 4000 lines all the same
Loading of module with unavailable key is rejected
Second try:-
This time running the update took over 10 mins.
Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
/opt/pakfire/db/core/mine contained 167 beforehand.
Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
Repository was set back to Stable.
Changed again to Testing and it said again that there was an update from 167 to 168.
Before running the pakfire directory had the following
-rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
After running it had
-rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
Rebooted.
All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
/opt/pakfire/db/core/mine has 168 in it
I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
Third try:-
-rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
Selected Testing and ran update from 167 to 168 which occurred very quickly.
-rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... May 11 13:59:34 ipfire pakfire: CLEANUP: tmp May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... May 11 14:00:31 ipfire pakfire: CLEANUP: tmp May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... May 11 14:00:31 ipfire pakfire: CLEANUP: tmp May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
May 11 14:01:10 ipfire pakfire: CLEANUP: tmp May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... May 11 14:01:10 ipfire pakfire: CLEANUP: tmp May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... May 11 14:01:11 ipfire pakfire: CLEANUP: tmp May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
Rebooted Lots of ipset restore error but went by too fast to get more detail. Several repeats of the following two lines on the console screen:-
iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) Perhaps iptables or your kernel needs to be upgraded.
Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
I have given up at this point.
I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
Regards,
Adolf.
Hi Adolf,
Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code: https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions....
But unfortunately this is all I can contribute, my test system updated to 168 without problems!
Best regards Leo
Am 11.05.2022 um 15:11 schrieb Adolf Belka:
Hi All,
I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
Normally my updates go without any real problems. This is the first time where this is not the case.
First try:-
After running the update, which went quite quickly, the bottom of the screen had
Core Update 167 Development Build: master/c22d834c
I couldn't remember if this is what is expected or not for Testing releases.
Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
Lots of ipset restore error but went by too fast to get more detail.
Tried to access via the WUI and no access. Tried via SSH and no access.
Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
starting DUID 00:04:70:........ red0: interface not found
So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
Looked in bootlog and this was filled with around 4000 lines all the same
Loading of module with unavailable key is rejected
Second try:-
This time running the update took over 10 mins.
Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
/opt/pakfire/db/core/mine contained 167 beforehand.
Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
Repository was set back to Stable.
Changed again to Testing and it said again that there was an update from 167 to 168.
Before running the pakfire directory had the following
-rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
After running it had
-rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
Rebooted.
All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
/opt/pakfire/db/core/mine has 168 in it
I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
Third try:-
-rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
Selected Testing and ran update from 167 to 168 which occurred very quickly.
-rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... May 11 13:59:34 ipfire pakfire: CLEANUP: tmp May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... May 11 14:00:31 ipfire pakfire: CLEANUP: tmp May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... May 11 14:00:31 ipfire pakfire: CLEANUP: tmp May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
May 11 14:01:10 ipfire pakfire: CLEANUP: tmp May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... May 11 14:01:10 ipfire pakfire: CLEANUP: tmp May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... May 11 14:01:11 ipfire pakfire: CLEANUP: tmp May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
Rebooted Lots of ipset restore error but went by too fast to get more detail. Several repeats of the following two lines on the console screen:-
iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) Perhaps iptables or your kernel needs to be upgraded.
Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
I have given up at this point.
I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
Regards,
Adolf.
Hi Leo,
On 11/05/2022 15:26, Leo Hofmann wrote:
Hi Adolf,
Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions....
Now I see it in the code. Thanks very much.
But unfortunately this is all I can contribute, my test system updated to 168 without problems!
Thanks for your help anyway. It's got rid of one of my questions.
Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. Will wait to hear other inputs on this problem. Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
Regards, Adolf.
Best regards Leo
Am 11.05.2022 um 15:11 schrieb Adolf Belka:
Hi All,
I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
Normally my updates go without any real problems. This is the first time where this is not the case.
First try:-
After running the update, which went quite quickly, the bottom of the screen had
Core Update 167 Development Build: master/c22d834c
I couldn't remember if this is what is expected or not for Testing releases.
Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
Lots of ipset restore error but went by too fast to get more detail.
Tried to access via the WUI and no access. Tried via SSH and no access.
Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
starting DUID 00:04:70:........ red0: interface not found
So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
Looked in bootlog and this was filled with around 4000 lines all the same
Loading of module with unavailable key is rejected
Second try:-
This time running the update took over 10 mins.
Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
/opt/pakfire/db/core/mine contained 167 beforehand.
Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
Repository was set back to Stable.
Changed again to Testing and it said again that there was an update from 167 to 168.
Before running the pakfire directory had the following
-rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
After running it had
-rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
Rebooted.
All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
/opt/pakfire/db/core/mine has 168 in it
I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
Third try:-
-rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
Selected Testing and ran update from 167 to 168 which occurred very quickly.
-rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... May 11 13:59:34 ipfire pakfire: CLEANUP: tmp May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... May 11 14:00:31 ipfire pakfire: CLEANUP: tmp May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... May 11 14:00:31 ipfire pakfire: CLEANUP: tmp May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
May 11 14:01:10 ipfire pakfire: CLEANUP: tmp May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... May 11 14:01:10 ipfire pakfire: CLEANUP: tmp May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... May 11 14:01:11 ipfire pakfire: CLEANUP: tmp May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
Rebooted Lots of ipset restore error but went by too fast to get more detail. Several repeats of the following two lines on the console screen:-
iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) Perhaps iptables or your kernel needs to be upgraded.
Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
I have given up at this point.
I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
Regards,
Adolf.
Hi All,
On 11/05/2022 16:00, Adolf Belka wrote:
Hi Leo,
On 11/05/2022 15:26, Leo Hofmann wrote:
Hi Adolf,
Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions....
Now I see it in the code. Thanks very much.
But unfortunately this is all I can contribute, my test system updated to 168 without problems!
Thanks for your help anyway. It's got rid of one of my questions.
Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. Will wait to hear other inputs on this problem. Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
Regards,
Adolf.
Regards, Adolf.
Best regards Leo
Am 11.05.2022 um 15:11 schrieb Adolf Belka:
Hi All,
I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
Normally my updates go without any real problems. This is the first time where this is not the case.
First try:-
After running the update, which went quite quickly, the bottom of the screen had
Core Update 167 Development Build: master/c22d834c
I couldn't remember if this is what is expected or not for Testing releases.
Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
Lots of ipset restore error but went by too fast to get more detail.
Tried to access via the WUI and no access. Tried via SSH and no access.
Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
starting DUID 00:04:70:........ red0: interface not found
So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
Looked in bootlog and this was filled with around 4000 lines all the same
Loading of module with unavailable key is rejected
Second try:-
This time running the update took over 10 mins.
Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
/opt/pakfire/db/core/mine contained 167 beforehand.
Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
Repository was set back to Stable.
Changed again to Testing and it said again that there was an update from 167 to 168.
Before running the pakfire directory had the following
-rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
After running it had
-rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
Rebooted.
All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
/opt/pakfire/db/core/mine has 168 in it
I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
Third try:-
-rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
Selected Testing and ran update from 167 to 168 which occurred very quickly.
-rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... May 11 13:59:34 ipfire pakfire: CLEANUP: tmp May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... May 11 14:00:31 ipfire pakfire: CLEANUP: tmp May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... May 11 14:00:31 ipfire pakfire: CLEANUP: tmp May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
May 11 14:01:10 ipfire pakfire: CLEANUP: tmp May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... May 11 14:01:10 ipfire pakfire: CLEANUP: tmp May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... May 11 14:01:11 ipfire pakfire: CLEANUP: tmp May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
Rebooted Lots of ipset restore error but went by too fast to get more detail. Several repeats of the following two lines on the console screen:-
iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) Perhaps iptables or your kernel needs to be upgraded.
Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
I have given up at this point.
I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
Regards,
Adolf.
I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c
Shouldn’t this be Core Update 168?
Jon
On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote:
Hi All,
On 11/05/2022 16:00, Adolf Belka wrote:
Hi Leo,
On 11/05/2022 15:26, Leo Hofmann wrote:
Hi Adolf,
Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762
Now I see it in the code. Thanks very much.
But unfortunately this is all I can contribute, my test system updated to 168 without problems!
Thanks for your help anyway. It's got rid of one of my questions.
Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. Will wait to hear other inputs on this problem. Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
Regards,
Adolf.
Regards, Adolf.
Best regards Leo
Am 11.05.2022 um 15:11 schrieb Adolf Belka:
Hi All,
I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
Normally my updates go without any real problems. This is the first time where this is not the case.
First try:-
After running the update, which went quite quickly, the bottom of the screen had
Core Update 167 Development Build: master/c22d834c
I couldn't remember if this is what is expected or not for Testing releases.
Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
Lots of ipset restore error but went by too fast to get more detail.
Tried to access via the WUI and no access. Tried via SSH and no access.
Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
starting DUID 00:04:70:........ red0: interface not found
So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
Looked in bootlog and this was filled with around 4000 lines all the same
Loading of module with unavailable key is rejected
Second try:-
This time running the update took over 10 mins.
Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
/opt/pakfire/db/core/mine contained 167 beforehand.
Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
Repository was set back to Stable.
Changed again to Testing and it said again that there was an update from 167 to 168.
Before running the pakfire directory had the following
-rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
After running it had
-rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
Rebooted.
All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
/opt/pakfire/db/core/mine has 168 in it
I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
Third try:-
-rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
Selected Testing and ran update from 167 to 168 which occurred very quickly.
-rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... May 11 13:59:34 ipfire pakfire: CLEANUP: tmp May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... May 11 14:00:31 ipfire pakfire: CLEANUP: tmp May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... May 11 14:00:31 ipfire pakfire: CLEANUP: tmp May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
May 11 14:01:10 ipfire pakfire: CLEANUP: tmp May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... May 11 14:01:10 ipfire pakfire: CLEANUP: tmp May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... May 11 14:01:11 ipfire pakfire: CLEANUP: tmp May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
Rebooted Lots of ipset restore error but went by too fast to get more detail. Several repeats of the following two lines on the console screen:-
iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) Perhaps iptables or your kernel needs to be upgraded.
Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
I have given up at this point.
I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
Regards,
Adolf.
Hi All,
On 11/05/2022 20:48, Jon Murphy wrote:
I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
*IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c*
Shouldn’t this be Core Update 168?
That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
I think I am going to raise a bug on this.
Regards, Adolf.
Jon
On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote:
Hi All,
On 11/05/2022 16:00, Adolf Belka wrote:
Hi Leo,
On 11/05/2022 15:26, Leo Hofmann wrote:
Hi Adolf,
Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762
Now I see it in the code. Thanks very much.
But unfortunately this is all I can contribute, my test system updated to 168 without problems!
Thanks for your help anyway. It's got rid of one of my questions.
Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. Will wait to hear other inputs on this problem. Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
Regards,
Adolf.
Regards, Adolf.
Best regards Leo
Am 11.05.2022 um 15:11 schrieb Adolf Belka:
Hi All,
I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
Normally my updates go without any real problems. This is the first time where this is not the case.
First try:-
After running the update, which went quite quickly, the bottom of the screen had
Core Update 167 Development Build: master/c22d834c
I couldn't remember if this is what is expected or not for Testing releases.
Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
Lots of ipset restore error but went by too fast to get more detail.
Tried to access via the WUI and no access. Tried via SSH and no access.
Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
starting DUID 00:04:70:........ red0: interface not found
So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
Looked in bootlog and this was filled with around 4000 lines all the same
Loading of module with unavailable key is rejected
Second try:-
This time running the update took over 10 mins.
Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
/opt/pakfire/db/core/mine contained 167 beforehand.
Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
Repository was set back to Stable.
Changed again to Testing and it said again that there was an update from 167 to 168.
Before running the pakfire directory had the following
-rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
After running it had
-rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
Rebooted.
All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
/opt/pakfire/db/core/mine has 168 in it
I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
Third try:-
-rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
Selected Testing and ran update from 167 to 168 which occurred very quickly.
-rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... May 11 13:59:34 ipfire pakfire: CLEANUP: tmp May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... May 11 14:00:31 ipfire pakfire: CLEANUP: tmp May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... May 11 14:00:31 ipfire pakfire: CLEANUP: tmp May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
May 11 14:01:10 ipfire pakfire: CLEANUP: tmp May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... May 11 14:01:10 ipfire pakfire: CLEANUP: tmp May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... May 11 14:01:11 ipfire pakfire: CLEANUP: tmp May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
Rebooted Lots of ipset restore error but went by too fast to get more detail. Several repeats of the following two lines on the console screen:-
iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) Perhaps iptables or your kernel needs to be upgraded.
Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
I have given up at this point.
I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
Regards,
Adolf.
Hello,
Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
On 11 May 2022, at 20:08, Adolf Belka adolf.belka@ipfire.org wrote:
Hi All,
On 11/05/2022 20:48, Jon Murphy wrote:
I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info: *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* Shouldn’t this be Core Update 168?
That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
That seems to be a bug that should not be there. Did we recently update apache?
-Michael
Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
I think I am going to raise a bug on this.
Regards, Adolf.
Jon
On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote:
Hi All,
On 11/05/2022 16:00, Adolf Belka wrote:
Hi Leo,
On 11/05/2022 15:26, Leo Hofmann wrote:
Hi Adolf,
Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762
Now I see it in the code. Thanks very much.
But unfortunately this is all I can contribute, my test system updated to 168 without problems!
Thanks for your help anyway. It's got rid of one of my questions.
Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. Will wait to hear other inputs on this problem. Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
Regards,
Adolf.
Regards, Adolf.
Best regards Leo
Am 11.05.2022 um 15:11 schrieb Adolf Belka:
Hi All,
I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
Normally my updates go without any real problems. This is the first time where this is not the case.
First try:-
After running the update, which went quite quickly, the bottom of the screen had
Core Update 167 Development Build: master/c22d834c
I couldn't remember if this is what is expected or not for Testing releases.
Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
Lots of ipset restore error but went by too fast to get more detail.
Tried to access via the WUI and no access. Tried via SSH and no access.
Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
starting DUID 00:04:70:........ red0: interface not found
So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
Looked in bootlog and this was filled with around 4000 lines all the same
Loading of module with unavailable key is rejected
Second try:-
This time running the update took over 10 mins.
Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
/opt/pakfire/db/core/mine contained 167 beforehand.
Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
Repository was set back to Stable.
Changed again to Testing and it said again that there was an update from 167 to 168.
Before running the pakfire directory had the following
-rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
After running it had
-rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
Rebooted.
All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
/opt/pakfire/db/core/mine has 168 in it
I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
Third try:-
-rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
Selected Testing and ran update from 167 to 168 which occurred very quickly.
-rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... May 11 13:59:34 ipfire pakfire: CLEANUP: tmp May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... May 11 14:00:31 ipfire pakfire: CLEANUP: tmp May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... May 11 14:00:31 ipfire pakfire: CLEANUP: tmp May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
May 11 14:01:10 ipfire pakfire: CLEANUP: tmp May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... May 11 14:01:10 ipfire pakfire: CLEANUP: tmp May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... May 11 14:01:11 ipfire pakfire: CLEANUP: tmp May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
Rebooted Lots of ipset restore error but went by too fast to get more detail. Several repeats of the following two lines on the console screen:-
iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) Perhaps iptables or your kernel needs to be upgraded.
Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
I have given up at this point.
I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
Regards,
Adolf.
On Thursday 12 May 2022 10:13 Michael Tremer wrote:
Hello,
Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
My Pakfire upgrade to 168 on my development APU2 board failed during upgrade and I lost ethernet communication with the PC.
The APU2 now fails after the grub prompt with the error:
*IPFire 2.27 (x86_64) - core166 Development Build: master/8f696f60 GNU/Linu
Loading Linux 5.15.23-ipfire ... error: file `/vmlinuz-5.15.23-ipfire' not found. Loading initial ramdisk ... error: you need to load the kernel first.
so it looks like update-initramfs didn't run after the upgrade.
I'll try to boot the box from a usbstick and see if I can access the disk.
Rob
Hello Rob,
You seem to already have lost the kernel there which is not part of 168.
Can you extract any log files from /var/log/pakfire?
-Michael
On 12 May 2022, at 11:43, Rob Brewer ipfire-devel@grantura.co.uk wrote:
On Thursday 12 May 2022 10:13 Michael Tremer wrote:
Hello,
Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
My Pakfire upgrade to 168 on my development APU2 board failed during upgrade and I lost ethernet communication with the PC.
The APU2 now fails after the grub prompt with the error:
*IPFire 2.27 (x86_64) - core166 Development Build: master/8f696f60 GNU/Linu
Loading Linux 5.15.23-ipfire ... error: file `/vmlinuz-5.15.23-ipfire' not found. Loading initial ramdisk ... error: you need to load the kernel first.
so it looks like update-initramfs didn't run after the upgrade.
I'll try to boot the box from a usbstick and see if I can access the disk.
Rob
On Thursday 12 May 2022 13:51 Michael Tremer wrote:
Hello Rob,
You seem to already have lost the kernel there which is not part of 168.
Can you extract any log files from /var/log/pakfire?
-Michael
I was upgrading 165-168 so I think Linux 5.15.23-ipfire was from 165. I am trying to build a bootable USB drive so I can mount the ssd and look at the Pakfire logs.
Rob
On 12 May 2022, at 11:43, Rob Brewer ipfire-devel@grantura.co.uk wrote:
On Thursday 12 May 2022 10:13 Michael Tremer wrote:
Hello,
Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
My Pakfire upgrade to 168 on my development APU2 board failed during upgrade and I lost ethernet communication with the PC.
The APU2 now fails after the grub prompt with the error:
*IPFire 2.27 (x86_64) - core166 Development Build: master/8f696f60 GNU/Linu
Loading Linux 5.15.23-ipfire ... error: file `/vmlinuz-5.15.23-ipfire' not found. Loading initial ramdisk ... error: you need to load the kernel first.
so it looks like update-initramfs didn't run after the upgrade.
I'll try to boot the box from a usbstick and see if I can access the disk.
Rob
On Thursday 12 May 2022 14:15 Rob Brewer wrote:
On Thursday 12 May 2022 13:51 Michael Tremer wrote:
Hello Rob,
You seem to already have lost the kernel there which is not part of 168.
Can you extract any log files from /var/log/pakfire?
-Michael
I was upgrading 165-168 so I think Linux 5.15.23-ipfire was from 165. I am trying to build a bootable USB drive so I can mount the ssd and look at the Pakfire logs.
Rob
I have been unable to boot from a usb drive so I re-fitted an old mSATA SSD with an earlier version of IPFIRE and was surprisingly unable to boot from there also.
On investigation the bios is reporting:
SeaBIOS (version rel-1.14.0.1-0-g8610266a)
yet I manually upgraded it to v4.16.0.2 a few weeks ago after a discussion with Bernard.
https://community.ipfire.org/t/entropy-with-new-apu-firmware/7707
Could something have downgraded or corrupted the BIOS?
I'll try to re-flash back to v4.16.0.2 if I can and try to boot again.
Rob
On 12 May 2022, at 11:43, Rob Brewer ipfire-devel@grantura.co.uk wrote:
On Thursday 12 May 2022 10:13 Michael Tremer wrote:
Hello,
Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
My Pakfire upgrade to 168 on my development APU2 board failed during upgrade and I lost ethernet communication with the PC.
The APU2 now fails after the grub prompt with the error:
*IPFire 2.27 (x86_64) - core166 Development Build: master/8f696f60 GNU/Linu
Loading Linux 5.15.23-ipfire ... error: file `/vmlinuz-5.15.23-ipfire' not found. Loading initial ramdisk ... error: you need to load the kernel first.
so it looks like update-initramfs didn't run after the upgrade.
I'll try to boot the box from a usbstick and see if I can access the disk.
Rob
On Friday 13 May 2022 16:02 Rob Brewer wrote:
On Thursday 12 May 2022 14:15 Rob Brewer wrote:
On Thursday 12 May 2022 13:51 Michael Tremer wrote:
Hello Rob,
You seem to already have lost the kernel there which is not part of 168.
Can you extract any log files from /var/log/pakfire?
-Michael
I was upgrading 165-168 so I think Linux 5.15.23-ipfire was from 165. I am trying to build a bootable USB drive so I can mount the ssd and look at the Pakfire logs.
Rob
I have been unable to boot from a usb drive so I re-fitted an old mSATA SSD with an earlier version of IPFIRE and was surprisingly unable to boot from there also.
On investigation the bios is reporting:
SeaBIOS (version rel-1.14.0.1-0-g8610266a)
yet I manually upgraded it to v4.16.0.2 a few weeks ago after a discussion with Bernard.
https://community.ipfire.org/t/entropy-with-new-apu-firmware/7707
Could something have downgraded or corrupted the BIOS?
I'll try to re-flash back to v4.16.0.2 if I can and try to boot again.
Rob
Update..
Sucessfully booted from usb disk.
There appears to be an error in grub.cfg as the menu is looking for the wrong ipfire.img file:
/boot/grub/grub.cfg
# # DO NOT EDIT THIS FILE # # It is automatically generated by grub-mkconfig using templates # from /etc/grub.d and settings from /etc/default/grub #
### BEGIN /etc/grub.d/00_cloud ### ### END /etc/grub.d/00_cloud ###
### BEGIN /etc/grub.d/00_header ### if [ -s $prefix/grubenv ]; then load_env fi if [ "${next_entry}" ] ; then set default="${next_entry}" set next_entry= save_env next_entry set boot_once=true else set default="${saved_entry}" fi
if [ x"${feature_menuentry_id}" = xy ]; then menuentry_id_option="--id" else menuentry_id_option="" fi
export menuentry_id_option
if [ "${prev_saved_entry}" ]; then set saved_entry="${prev_saved_entry}" save_env saved_entry set prev_saved_entry= save_env prev_saved_entry set boot_once=true fi
function savedefault { if [ -z "${boot_once}" ]; then saved_entry="${chosen}" save_env saved_entry fi }
function load_video { if [ x$feature_all_video_module = xy ]; then insmod all_video else insmod efi_gop insmod efi_uga insmod ieee1275_fb insmod vbe insmod vga insmod video_bochs insmod video_cirrus fi }
serial --unit=0 --speed=115200 terminal_input serial terminal_output serial if [ x$feature_timeout_style = xy ] ; then set timeout_style=menu set timeout=5 # Fallback normal timeout code in case the timeout_style feature is # unavailable. else set timeout=5 fi ### END /etc/grub.d/00_header ###
### BEGIN /etc/grub.d/10_linux ### menuentry 'IPFire 2.27 (x86_64) - core166 Development Build: master/8f696f60 GNU/Linux' --class ipfire --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-1c836493-be8b-401c-ad0e-fb9ef6e85328' { load_video set gfxpayload=keep insmod gzio insmod part_msdos insmod ext2 set root='hd0,msdos1' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 257d45b3-1483-44ab-815c-029ffa1bc8f6 else search --no-floppy --fs-uuid --set=root 257d45b3-1483-44ab-815c-029ffa1bc8f6 fi echo 'Loading Linux 5.15.23-ipfire ...' linux /vmlinuz-5.15.23-ipfire root=UUID=1c836493-be8b-401c-ad0e-fb9ef6e85328 ro panic=10 console=ttyS0,115200n8 echo 'Loading initial ramdisk ...' initrd /initramfs-5.15.23-ipfire.img } submenu 'Advanced options for IPFire 2.27 (x86_64) - core166 Development Build: master/8f696f60 GNU/Linux' $menuentry_id_option 'gnulinux-advanced-1c836493-be8b-401c-ad0e-fb9ef6e85328' { menuentry 'IPFire 2.27 (x86_64) - core166 Development Build: master/8f696f60 GNU/Linux, with Linux 5.15.23-ipfire' --class ipfire --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.15.23-ipfire-advan load_video set gfxpayload=keep insmod gzio insmod part_msdos insmod ext2 set root='hd0,msdos1' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 257d45b3-1483-44ab-815c-029ffa1bc8f6 else search --no-floppy --fs-uuid --set=root 257d45b3-1483-44ab-815c-029ffa1bc8f6 fi echo 'Loading Linux 5.15.23-ipfire ...' linux /vmlinuz-5.15.23-ipfire root=UUID=1c836493-be8b-401c-ad0e-fb9ef6e85328 ro panic=10 console=ttyS0,115200n8 echo 'Loading initial ramdisk ...' initrd /initramfs-5.15.23-ipfire.img } }
### END /etc/grub.d/10_linux ###
### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###
### BEGIN /etc/grub.d/30_os-prober ### ### END /etc/grub.d/30_os-prober ###
### BEGIN /etc/grub.d/30_uefi-firmware ### ### END /etc/grub.d/30_uefi-firmware ###
### BEGIN /etc/grub.d/40_custom ### # This file provides an easy way to add custom menu entries. Simply type the # menu entries you want to add after this comment. Be careful not to change # the 'exec tail' line above. ### END /etc/grub.d/40_custom ###
### BEGIN /etc/grub.d/41_custom ### if [ -f ${config_directory}/custom.cfg ]; then source ${config_directory}/custom.cfg elif [ -z "${config_directory}" -a -f $prefix/custom.cfg ]; then source $prefix/custom.cfg fi
==================================
ls- -l /boot
-rw-r--r-- 1 root root 4869786 May 3 02:02 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 182471 May 3 02:02 config-5.15.35-ipfire drwxr-xr-x 2 root root 4096 Mar 1 12:55 efi drwxr-xr-x 6 root root 4096 May 11 07:59 grub drwx------ 2 root root 16384 Mar 1 12:55 lost+found -rw-r--r-- 1 root root 6993120 May 3 02:02 vmlinuz-5.15.35-ipfire
Grub should be looking for 5.15.35-ipfire but is incorrectly looking for 5.15.23-ipfire
========================================
root@box:~# ls -l /mnt/var/log/pakfire total 1872 -rw-r--r-- 1 root root 422 May 7 09:29 install-cpufrequtils.log -rw-r--r-- 1 root root 73 Mar 29 15:53 install-firmware-update.log -rw-r--r-- 1 root root 66 Mar 29 15:53 install-flashrom.log -rw-r--r-- 1 root root 1033 Mar 10 18:00 install-nano.log -rw-r--r-- 1 root root 380 Mar 29 15:53 install-pcengines-apu-firmware.log -rw-r--r-- 1 root root 79 Mar 29 15:52 install-rsync.log -rw-r--r-- 1 root root 420549 Mar 29 12:14 update-core-upgrade-163.log -rw-r--r-- 1 root root 289507 Mar 29 12:14 update-core-upgrade-164.log -rw-r--r-- 1 root root 841876 May 11 07:59 update-core-upgrade-165.log -rw-r--r-- 1 root root 1718 May 11 07:59 update-core-upgrade-166.log -rw-r--r-- 1 root root 326352 May 11 08:00 update-core-upgrade-167.log -rw-r--r-- 1 root root 2564 Mar 29 12:16 update-nano.log
Rob
On 12 May 2022, at 11:43, Rob Brewer ipfire-devel@grantura.co.uk wrote:
On Thursday 12 May 2022 10:13 Michael Tremer wrote:
Hello,
Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
My Pakfire upgrade to 168 on my development APU2 board failed during upgrade and I lost ethernet communication with the PC.
The APU2 now fails after the grub prompt with the error:
*IPFire 2.27 (x86_64) - core166 Development Build: master/8f696f60 GNU/Linu
Loading Linux 5.15.23-ipfire ... error: file `/vmlinuz-5.15.23-ipfire' not found. Loading initial ramdisk ... error: you need to load the kernel first.
so it looks like update-initramfs didn't run after the upgrade.
I'll try to boot the box from a usbstick and see if I can access the disk.
Rob
Hi Rob,
I don’t thing the "SeaBIOS (version rel-1.14.0.1-0-g8610266a)" equals an old BIOS. I see the same SeaBIOS line except I am on "BIOS version v4.15.0.1 "
I cannot reboot now to double check (people working from home). So this is from an old log file.
03-11 20:32:39.779: |*IPFire 2.27 (x86_64) - core163 GNU/Linux, with Linux 5.15.6-ipfire PC Engines apu4 03-11 20:32:39.779: coreboot build 20212311 | 03-11 20:32:39.779: BIOS version v4.15.0.1 | 03-11 20:32:39.779: 4080 MB ECC DRAM | 03-11 20:32:39.779: | | 03-11 20:33:00.772: SeaBIOS (version rel-1.14.0.1-0-g8610266a)
03-11 20:33:03.781: Press F10 key now for boot menu
Jon
On May 13, 2022, at 10:02 AM, Rob Brewer ipfire-devel@grantura.co.uk wrote:
On Thursday 12 May 2022 14:15 Rob Brewer wrote:
On Thursday 12 May 2022 13:51 Michael Tremer wrote:
Hello Rob,
You seem to already have lost the kernel there which is not part of 168.
Can you extract any log files from /var/log/pakfire?
-Michael
I was upgrading 165-168 so I think Linux 5.15.23-ipfire was from 165. I am trying to build a bootable USB drive so I can mount the ssd and look at the Pakfire logs.
Rob
I have been unable to boot from a usb drive so I re-fitted an old mSATA SSD with an earlier version of IPFIRE and was surprisingly unable to boot from there also.
On investigation the bios is reporting:
SeaBIOS (version rel-1.14.0.1-0-g8610266a)
yet I manually upgraded it to v4.16.0.2 a few weeks ago after a discussion with Bernard.
https://community.ipfire.org/t/entropy-with-new-apu-firmware/7707
Could something have downgraded or corrupted the BIOS?
I'll try to re-flash back to v4.16.0.2 if I can and try to boot again.
Rob
On 12 May 2022, at 11:43, Rob Brewer ipfire-devel@grantura.co.uk wrote:
On Thursday 12 May 2022 10:13 Michael Tremer wrote:
Hello,
Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
My Pakfire upgrade to 168 on my development APU2 board failed during upgrade and I lost ethernet communication with the PC.
The APU2 now fails after the grub prompt with the error:
*IPFire 2.27 (x86_64) - core166 Development Build: master/8f696f60 GNU/Linu
Loading Linux 5.15.23-ipfire ... error: file `/vmlinuz-5.15.23-ipfire' not found. Loading initial ramdisk ... error: you need to load the kernel first.
so it looks like update-initramfs didn't run after the upgrade.
I'll try to boot the box from a usbstick and see if I can access the disk.
Rob
Hello,
On 13 May 2022, at 16:02, Rob Brewer ipfire-devel@grantura.co.uk wrote:
On Thursday 12 May 2022 14:15 Rob Brewer wrote:
On Thursday 12 May 2022 13:51 Michael Tremer wrote:
Hello Rob,
You seem to already have lost the kernel there which is not part of 168.
Can you extract any log files from /var/log/pakfire?
-Michael
I was upgrading 165-168 so I think Linux 5.15.23-ipfire was from 165. I am trying to build a bootable USB drive so I can mount the ssd and look at the Pakfire logs.
Rob
I have been unable to boot from a usb drive so I re-fitted an old mSATA SSD with an earlier version of IPFIRE and was surprisingly unable to boot from there also.
On investigation the bios is reporting:
SeaBIOS (version rel-1.14.0.1-0-g8610266a)
yet I manually upgraded it to v4.16.0.2 a few weeks ago after a discussion with Bernard.
https://community.ipfire.org/t/entropy-with-new-apu-firmware/7707
Could something have downgraded or corrupted the BIOS?
No, we never touch this as it is too unpredictable what would happen.
I'll try to re-flash back to v4.16.0.2 if I can and try to boot again.
But that is not the same version number that you are comparing there.
SeaBIOS is a part of the firmware and comes in a different version.
Rob
On 12 May 2022, at 11:43, Rob Brewer ipfire-devel@grantura.co.uk wrote:
On Thursday 12 May 2022 10:13 Michael Tremer wrote:
Hello,
Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
My Pakfire upgrade to 168 on my development APU2 board failed during upgrade and I lost ethernet communication with the PC.
The APU2 now fails after the grub prompt with the error:
*IPFire 2.27 (x86_64) - core166 Development Build: master/8f696f60 GNU/Linu
Loading Linux 5.15.23-ipfire ... error: file `/vmlinuz-5.15.23-ipfire' not found. Loading initial ramdisk ... error: you need to load the kernel first.
so it looks like update-initramfs didn't run after the upgrade.
I'll try to boot the box from a usbstick and see if I can access the disk.
Rob
On Friday 13 May 2022 21:36 Michael Tremer wrote:
Hello,
On 13 May 2022, at 16:02, Rob Brewer ipfire-devel@grantura.co.uk wrote:
On Thursday 12 May 2022 14:15 Rob Brewer wrote:
On Thursday 12 May 2022 13:51 Michael Tremer wrote:
Hello Rob,
You seem to already have lost the kernel there which is not part of 168.
Can you extract any log files from /var/log/pakfire?
-Michael
I was upgrading 165-168 so I think Linux 5.15.23-ipfire was from 165. I am trying to build a bootable USB drive so I can mount the ssd and look at the Pakfire logs.
Rob
I have been unable to boot from a usb drive so I re-fitted an old mSATA SSD with an earlier version of IPFIRE and was surprisingly unable to boot from there also.
On investigation the bios is reporting:
SeaBIOS (version rel-1.14.0.1-0-g8610266a)
yet I manually upgraded it to v4.16.0.2 a few weeks ago after a discussion with Bernard.
https://community.ipfire.org/t/entropy-with-new-apu-firmware/7707
Could something have downgraded or corrupted the BIOS?
No, we never touch this as it is too unpredictable what would happen.
I'll try to re-flash back to v4.16.0.2 if I can and try to boot again.
But that is not the same version number that you are comparing there.
SeaBIOS is a part of the firmware and comes in a different version.
Yes sorry the BIOS version was a red-herring. When I eventually could boot from USB I could see that the correct BIOS version was displayed on the boot screen.
My problem seems to be that grub hasn't been correctly updated to load the current kernel.
Rob
On Friday 13 May 2022 22:18 Rob Brewer wrote:
On Friday 13 May 2022 21:36 Michael Tremer wrote:
Hello,
On 13 May 2022, at 16:02, Rob Brewer ipfire-devel@grantura.co.uk wrote:
On Thursday 12 May 2022 14:15 Rob Brewer wrote:
On Thursday 12 May 2022 13:51 Michael Tremer wrote:
Hello Rob,
You seem to already have lost the kernel there which is not part of 168.
Can you extract any log files from /var/log/pakfire?
-Michael
I was upgrading 165-168 so I think Linux 5.15.23-ipfire was from 165. I am trying to build a bootable USB drive so I can mount the ssd and look at the Pakfire logs.
Rob
I have been unable to boot from a usb drive so I re-fitted an old mSATA SSD with an earlier version of IPFIRE and was surprisingly unable to boot from there also.
On investigation the bios is reporting:
SeaBIOS (version rel-1.14.0.1-0-g8610266a)
yet I manually upgraded it to v4.16.0.2 a few weeks ago after a discussion with Bernard.
https://community.ipfire.org/t/entropy-with-new-apu-firmware/7707
Could something have downgraded or corrupted the BIOS?
No, we never touch this as it is too unpredictable what would happen.
I'll try to re-flash back to v4.16.0.2 if I can and try to boot again.
But that is not the same version number that you are comparing there.
SeaBIOS is a part of the firmware and comes in a different version.
Yes sorry the BIOS version was a red-herring. When I eventually could boot from USB I could see that the correct BIOS version was displayed on the boot screen.
My problem seems to be that grub hasn't been correctly updated to load the current kernel.
On checking the pakfire log files it would seem that grub wasn't updated after kernel 5.15.35 was introduced in core 167.
I suspect this was caused by a failure in 'dracut' :
tail update-core-upgrade-167.log
dracut: Executing: /usr/bin/dracut --kver=5.15.35-ipfire --force dracut: *** Including module: modsign *** dracut: *** Including module: i18n *** dracut: *** Including module: dm *** dracut: Skipping udev rule: 64-device-mapper.rules dracut: Skipping udev rule: 60-persistent-storage-dm.rules dracut: Skipping udev rule: 55-dm.rules dracut: *** Including module: kernel-modules ***
I presume this should have updated grub but dracut is new to me so I'm not sure what to expect here!
Rob
Hi,
On 12/05/2022 11:13, Michael Tremer wrote:
Hello,
Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
Tried modprobe e1000 but this came back with the following error modprobe: ERROR: could not insert 'e1000': Key was rejected by service
So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
Regards Adolf
On 11 May 2022, at 20:08, Adolf Belka adolf.belka@ipfire.org wrote:
Hi All,
On 11/05/2022 20:48, Jon Murphy wrote:
I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info: *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* Shouldn’t this be Core Update 168?
That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
That seems to be a bug that should not be there. Did we recently update apache?
-Michael
Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
I think I am going to raise a bug on this.
Regards, Adolf.
Jon
On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote:
Hi All,
On 11/05/2022 16:00, Adolf Belka wrote:
Hi Leo,
On 11/05/2022 15:26, Leo Hofmann wrote:
Hi Adolf,
Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762
Now I see it in the code. Thanks very much.
But unfortunately this is all I can contribute, my test system updated to 168 without problems!
Thanks for your help anyway. It's got rid of one of my questions.
Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. Will wait to hear other inputs on this problem. Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
Regards,
Adolf.
Regards, Adolf.
Best regards Leo
Am 11.05.2022 um 15:11 schrieb Adolf Belka: > Hi All, > > I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts. > > Normally my updates go without any real problems. This is the first time where this is not the case. > > > First try:- > > After running the update, which went quite quickly, the bottom of the screen had > > Core Update 167 Development Build: master/c22d834c > > I couldn't remember if this is what is expected or not for Testing releases. > > Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped. > > Lots of ipset restore error but went by too fast to get more detail. > > Tried to access via the WUI and no access. Tried via SSH and no access. > > Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message > > starting > DUID 00:04:70:........ > red0: interface not found > > So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0. > > Looked in bootlog and this was filled with around 4000 lines all the same > > Loading of module with unavailable key is rejected > > > Second try:- > > This time running the update took over 10 mins. > > Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167. > > /opt/pakfire/db/core/mine contained 167 beforehand. > > Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time. > > Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28 > > Repository was set back to Stable. > > Changed again to Testing and it said again that there was an update from 167 to 168. > > Before running the pakfire directory had the following > > -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log > -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log > > After running it had > > -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log > -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log > -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log > > back to Core Update 167 Development Build: master/c22d834c at bottom of screen. > > Rebooted. > > All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c > > /opt/pakfire/db/core/mine has 168 in it > > I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168. > > > Third try:- > > -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log > > Selected Testing and ran update from 167 to 168 which occurred very quickly. > > -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log > -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log > > Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell. > > May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! > May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce > May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 > May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire > May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list > May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire > May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes > May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK > May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... > May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. > May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire > May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 > May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list > May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 > May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes > May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK > May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... > May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. > May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 > May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire > May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list > May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire > May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes > May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK > May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... > May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. > May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire > May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... > May 11 13:59:34 ipfire pakfire: CLEANUP: tmp > May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 > May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 > May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... > May 11 14:00:31 ipfire pakfire: CLEANUP: tmp > May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. > May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... > May 11 14:00:31 ipfire pakfire: CLEANUP: tmp > May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 > May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 > May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts... > > > May 11 14:01:10 ipfire pakfire: CLEANUP: tmp > May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. > May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce > May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... > May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. > May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire > May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list > May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire > May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes > May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK > May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... > May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. > May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire > May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... > May 11 14:01:10 ipfire pakfire: CLEANUP: tmp > May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio > May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 > May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... > May 11 14:01:11 ipfire pakfire: CLEANUP: tmp > May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. > May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing. > > Rebooted > Lots of ipset restore error but went by too fast to get more detail. > Several repeats of the following two lines on the console screen:- > > iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) > Perhaps iptables or your kernel needs to be upgraded. > > > Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present. > > > I have given up at this point. > > I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know. > > If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this. > > Regards, > > Adolf. >
Hello,
On 12 May 2022, at 12:25, Adolf Belka adolf.belka@ipfire.org wrote:
Hi,
On 12/05/2022 11:13, Michael Tremer wrote:
Hello, Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while. So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved. This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started? The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
Okay. That is indeed quite bad then.
Since there is no kernel in 168, is there a chance that we broke the update to 167?
Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
Third time lucky isn’t good enough for me :)
I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
Have there been any modules loaded? If one loads, they all should load.
Tried modprobe e1000 but this came back with the following error modprobe: ERROR: could not insert 'e1000': Key was rejected by service
So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
Regards Adolf
On 11 May 2022, at 20:08, Adolf Belka adolf.belka@ipfire.org wrote:
Hi All,
On 11/05/2022 20:48, Jon Murphy wrote:
I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info: *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* Shouldn’t this be Core Update 168?
That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
That seems to be a bug that should not be there. Did we recently update apache? -Michael
Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
I think I am going to raise a bug on this.
Regards, Adolf.
Jon
On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote:
Hi All,
On 11/05/2022 16:00, Adolf Belka wrote:
Hi Leo,
On 11/05/2022 15:26, Leo Hofmann wrote: > Hi Adolf, > > Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code: So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that. > https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 Now I see it in the code. Thanks very much. > > But unfortunately this is all I can contribute, my test system updated to 168 without problems! Thanks for your help anyway. It's got rid of one of my questions.
Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. Will wait to hear other inputs on this problem. Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
Regards,
Adolf.
Regards, Adolf.
> > Best regards > Leo > > Am 11.05.2022 um 15:11 schrieb Adolf Belka: >> Hi All, >> >> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts. >> >> Normally my updates go without any real problems. This is the first time where this is not the case. >> >> >> First try:- >> >> After running the update, which went quite quickly, the bottom of the screen had >> >> Core Update 167 Development Build: master/c22d834c >> >> I couldn't remember if this is what is expected or not for Testing releases. >> >> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped. >> >> Lots of ipset restore error but went by too fast to get more detail. >> >> Tried to access via the WUI and no access. Tried via SSH and no access. >> >> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message >> >> starting >> DUID 00:04:70:........ >> red0: interface not found >> >> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0. >> >> Looked in bootlog and this was filled with around 4000 lines all the same >> >> Loading of module with unavailable key is rejected >> >> >> Second try:- >> >> This time running the update took over 10 mins. >> >> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167. >> >> /opt/pakfire/db/core/mine contained 167 beforehand. >> >> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time. >> >> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28 >> >> Repository was set back to Stable. >> >> Changed again to Testing and it said again that there was an update from 167 to 168. >> >> Before running the pakfire directory had the following >> >> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >> >> After running it had >> >> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >> >> back to Core Update 167 Development Build: master/c22d834c at bottom of screen. >> >> Rebooted. >> >> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c >> >> /opt/pakfire/db/core/mine has 168 in it >> >> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168. >> >> >> Third try:- >> >> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >> >> Selected Testing and ran update from 167 to 168 which occurred very quickly. >> >> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >> >> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell. >> >> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! >> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce >> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 >> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire >> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. >> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 >> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes >> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. >> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire >> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. >> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... >> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 >> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... >> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. >> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... >> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 >> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts... >> >> >> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. >> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce >> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... >> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. >> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire >> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. >> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... >> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing. >> >> Rebooted >> Lots of ipset restore error but went by too fast to get more detail. >> Several repeats of the following two lines on the console screen:- >> >> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) >> Perhaps iptables or your kernel needs to be upgraded. >> >> >> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present. >> >> >> I have given up at this point. >> >> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know. >> >> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this. >> >> Regards, >> >> Adolf. >>
Hi Michael,
On 12/05/2022 14:53, Michael Tremer wrote:
Hello,
On 12 May 2022, at 12:25, Adolf Belka adolf.belka@ipfire.org wrote:
Hi,
On 12/05/2022 11:13, Michael Tremer wrote:
Hello, Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while. So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved. This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started? The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
Okay. That is indeed quite bad then.
Since there is no kernel in 168, is there a chance that we broke the update to 167?
I don't know but as far as I have been concerned CU167 has worked well on my vm. Is there something I should look for on my CU167 vm?
Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
Third time lucky isn’t good enough for me :)
Me neither :-)
I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
Have there been any modules loaded? If one loads, they all should load.
This is what I get with lsmod
Module Size Used by crct10dif_pclmul 16384 1 crc32_pclmul 16384 0 ghash_clmulni_intel 16384 0 serio_raw 20480 0 ohci_pci 20480 0 ata_generic 16384 0 pata_acpi 16384 0 video 57344 0
On my running CU1678 vm lsmod gives the following
Module Size Used by tun 61440 2 nfnetlink_queue 28672 1 xt_NFQUEUE 16384 8 xt_MASQUERADE 20480 1 cfg80211 1036288 0 rfkill 32768 1 cfg80211 8021q 40960 0 garp 16384 1 8021q xt_set 16384 260 ip_set_hash_net 49152 258 ip_set 57344 2 xt_set,ip_set_hash_net xt_hashlimit 20480 2 xt_multiport 20480 4 xt_policy 16384 5 xt_TCPMSS 16384 1 xt_conntrack 16384 7 xt_comment 16384 18 ipt_REJECT 16384 1 nf_reject_ipv4 16384 1 ipt_REJECT xt_LOG 20480 26 xt_limit 16384 25 xt_mark 16384 27 xt_connmark 16384 2 nf_log_syslog 24576 26 iptable_raw 16384 0 iptable_mangle 16384 1 iptable_filter 16384 1 vfat 24576 1 fat 90112 1 vfat sch_cake 36864 4 intel_powerclamp 20480 0 psmouse 184320 0 pcspkr 16384 0 i2c_piix4 28672 0 e1000 163840 0 i2c_core 106496 2 psmouse,i2c_piix4 crct10dif_pclmul 16384 1 crc32_pclmul 16384 0 ata_generic 16384 0 pata_acpi 16384 0 ghash_clmulni_intel 16384 0 serio_raw 20480 0 ohci_pci 20480 0 video 57344 0
The last 8 entries are the same but a whole load of others are missing.
Regards, Adolf.
Tried modprobe e1000 but this came back with the following error modprobe: ERROR: could not insert 'e1000': Key was rejected by service
So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
Regards Adolf
On 11 May 2022, at 20:08, Adolf Belka adolf.belka@ipfire.org wrote:
Hi All,
On 11/05/2022 20:48, Jon Murphy wrote:
I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info: *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* Shouldn’t this be Core Update 168?
That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
That seems to be a bug that should not be there. Did we recently update apache? -Michael
Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
I think I am going to raise a bug on this.
Regards, Adolf.
Jon
On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote:
Hi All,
On 11/05/2022 16:00, Adolf Belka wrote: > Hi Leo, > > On 11/05/2022 15:26, Leo Hofmann wrote: >> Hi Adolf, >> >> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code: > So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that. >> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 > Now I see it in the code. Thanks very much. >> >> But unfortunately this is all I can contribute, my test system updated to 168 without problems! > Thanks for your help anyway. It's got rid of one of my questions. > > Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. > Will wait to hear other inputs on this problem. > Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again. > Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
Regards,
Adolf.
> Regards, > Adolf. > >> >> Best regards >> Leo >> >> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>> Hi All, >>> >>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts. >>> >>> Normally my updates go without any real problems. This is the first time where this is not the case. >>> >>> >>> First try:- >>> >>> After running the update, which went quite quickly, the bottom of the screen had >>> >>> Core Update 167 Development Build: master/c22d834c >>> >>> I couldn't remember if this is what is expected or not for Testing releases. >>> >>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped. >>> >>> Lots of ipset restore error but went by too fast to get more detail. >>> >>> Tried to access via the WUI and no access. Tried via SSH and no access. >>> >>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message >>> >>> starting >>> DUID 00:04:70:........ >>> red0: interface not found >>> >>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0. >>> >>> Looked in bootlog and this was filled with around 4000 lines all the same >>> >>> Loading of module with unavailable key is rejected >>> >>> >>> Second try:- >>> >>> This time running the update took over 10 mins. >>> >>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167. >>> >>> /opt/pakfire/db/core/mine contained 167 beforehand. >>> >>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time. >>> >>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28 >>> >>> Repository was set back to Stable. >>> >>> Changed again to Testing and it said again that there was an update from 167 to 168. >>> >>> Before running the pakfire directory had the following >>> >>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>> >>> After running it had >>> >>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >>> >>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen. >>> >>> Rebooted. >>> >>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c >>> >>> /opt/pakfire/db/core/mine has 168 in it >>> >>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168. >>> >>> >>> Third try:- >>> >>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>> >>> Selected Testing and ran update from 167 to 168 which occurred very quickly. >>> >>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >>> >>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell. >>> >>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! >>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce >>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 >>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire >>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. >>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 >>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes >>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. >>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire >>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. >>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... >>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 >>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... >>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. >>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... >>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 >>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts... >>> >>> >>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. >>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce >>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... >>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. >>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire >>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. >>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... >>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing. >>> >>> Rebooted >>> Lots of ipset restore error but went by too fast to get more detail. >>> Several repeats of the following two lines on the console screen:- >>> >>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) >>> Perhaps iptables or your kernel needs to be upgraded. >>> >>> >>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present. >>> >>> >>> I have given up at this point. >>> >>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know. >>> >>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this. >>> >>> Regards, >>> >>> Adolf. >>>
Hello,
On 12 May 2022, at 21:10, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 12/05/2022 14:53, Michael Tremer wrote:
Hello,
On 12 May 2022, at 12:25, Adolf Belka adolf.belka@ipfire.org wrote:
Hi,
On 12/05/2022 11:13, Michael Tremer wrote:
Hello, Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while. So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved. This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started? The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
Okay. That is indeed quite bad then. Since there is no kernel in 168, is there a chance that we broke the update to 167?
I don't know but as far as I have been concerned CU167 has worked well on my vm. Is there something I should look for on my CU167 vm?
Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
Third time lucky isn’t good enough for me :)
Me neither :-)
I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
Have there been any modules loaded? If one loads, they all should load.
This is what I get with lsmod
Module Size Used by crct10dif_pclmul 16384 1 crc32_pclmul 16384 0 ghash_clmulni_intel 16384 0 serio_raw 20480 0 ohci_pci 20480 0 ata_generic 16384 0 pata_acpi 16384 0 video 57344 0
It looks like these modules could have come from the initramdisk image. At least that is working well.
On my running CU1678 vm lsmod gives the following
Module Size Used by tun 61440 2 nfnetlink_queue 28672 1 xt_NFQUEUE 16384 8 xt_MASQUERADE 20480 1 cfg80211 1036288 0 rfkill 32768 1 cfg80211 8021q 40960 0 garp 16384 1 8021q xt_set 16384 260 ip_set_hash_net 49152 258 ip_set 57344 2 xt_set,ip_set_hash_net xt_hashlimit 20480 2 xt_multiport 20480 4 xt_policy 16384 5 xt_TCPMSS 16384 1 xt_conntrack 16384 7 xt_comment 16384 18 ipt_REJECT 16384 1 nf_reject_ipv4 16384 1 ipt_REJECT xt_LOG 20480 26 xt_limit 16384 25 xt_mark 16384 27 xt_connmark 16384 2 nf_log_syslog 24576 26 iptable_raw 16384 0 iptable_mangle 16384 1 iptable_filter 16384 1 vfat 24576 1 fat 90112 1 vfat sch_cake 36864 4 intel_powerclamp 20480 0 psmouse 184320 0 pcspkr 16384 0 i2c_piix4 28672 0 e1000 163840 0 i2c_core 106496 2 psmouse,i2c_piix4 crct10dif_pclmul 16384 1 crc32_pclmul 16384 0 ata_generic 16384 0 pata_acpi 16384 0 ghash_clmulni_intel 16384 0 serio_raw 20480 0 ohci_pci 20480 0 video 57344 0
Yes, this looks more like it.
The last 8 entries are the same but a whole load of others are missing.
Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
-Michael
Regards, Adolf.
Tried modprobe e1000 but this came back with the following error modprobe: ERROR: could not insert 'e1000': Key was rejected by service
So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
Regards Adolf
On 11 May 2022, at 20:08, Adolf Belka adolf.belka@ipfire.org wrote:
Hi All,
On 11/05/2022 20:48, Jon Murphy wrote:
I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info: *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* Shouldn’t this be Core Update 168?
That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
That seems to be a bug that should not be there. Did we recently update apache? -Michael
Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
I think I am going to raise a bug on this.
Regards, Adolf.
Jon > On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote: > > Hi All, > > On 11/05/2022 16:00, Adolf Belka wrote: >> Hi Leo, >> >> On 11/05/2022 15:26, Leo Hofmann wrote: >>> Hi Adolf, >>> >>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code: >> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that. >>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 >> Now I see it in the code. Thanks very much. >>> >>> But unfortunately this is all I can contribute, my test system updated to 168 without problems! >> Thanks for your help anyway. It's got rid of one of my questions. >> >> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. >> Will wait to hear other inputs on this problem. >> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again. >> > Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones. > > I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm. > > When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/ > > Regards, > > Adolf. > >> Regards, >> Adolf. >> >>> >>> Best regards >>> Leo >>> >>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>> Hi All, >>>> >>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts. >>>> >>>> Normally my updates go without any real problems. This is the first time where this is not the case. >>>> >>>> >>>> First try:- >>>> >>>> After running the update, which went quite quickly, the bottom of the screen had >>>> >>>> Core Update 167 Development Build: master/c22d834c >>>> >>>> I couldn't remember if this is what is expected or not for Testing releases. >>>> >>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped. >>>> >>>> Lots of ipset restore error but went by too fast to get more detail. >>>> >>>> Tried to access via the WUI and no access. Tried via SSH and no access. >>>> >>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message >>>> >>>> starting >>>> DUID 00:04:70:........ >>>> red0: interface not found >>>> >>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0. >>>> >>>> Looked in bootlog and this was filled with around 4000 lines all the same >>>> >>>> Loading of module with unavailable key is rejected >>>> >>>> >>>> Second try:- >>>> >>>> This time running the update took over 10 mins. >>>> >>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167. >>>> >>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>> >>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time. >>>> >>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28 >>>> >>>> Repository was set back to Stable. >>>> >>>> Changed again to Testing and it said again that there was an update from 167 to 168. >>>> >>>> Before running the pakfire directory had the following >>>> >>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>> >>>> After running it had >>>> >>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >>>> >>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen. >>>> >>>> Rebooted. >>>> >>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c >>>> >>>> /opt/pakfire/db/core/mine has 168 in it >>>> >>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168. >>>> >>>> >>>> Third try:- >>>> >>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>> >>>> Selected Testing and ran update from 167 to 168 which occurred very quickly. >>>> >>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >>>> >>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell. >>>> >>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! >>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce >>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 >>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire >>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. >>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 >>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes >>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. >>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire >>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. >>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... >>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 >>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... >>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. >>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... >>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 >>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts... >>>> >>>> >>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. >>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce >>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... >>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. >>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire >>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. >>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... >>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing. >>>> >>>> Rebooted >>>> Lots of ipset restore error but went by too fast to get more detail. >>>> Several repeats of the following two lines on the console screen:- >>>> >>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) >>>> Perhaps iptables or your kernel needs to be upgraded. >>>> >>>> >>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present. >>>> >>>> >>>> I have given up at this point. >>>> >>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know. >>>> >>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this. >>>> >>>> Regards, >>>> >>>> Adolf.
Hi Michael,
I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
However I did copy the boot directory info for these two cases and found an interesting effect.
For the problem vm here was the boot directory layout of the CU167 base machine
drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire drwxr-xr-x 3 root root 16K Jan 1 1970 efi drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
After upgrade and before reboot this had changed to
drwxr-xr-x 5 root root 4.0K May 13 15:40 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K May 3 04:02 config-5.15.35-ipfire drwxr-xr-x 3 root root 16K Jan 1 1970 efi drwxr-xr-x 6 root root 4.0K May 13 15:41 grub -rw------- 1 root root 26M May 13 15:41 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M May 3 04:02 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M May 3 04:02 vmlinuz-5.15.35-ipfire
Dates and times of some of the files/directories have changed as would be expected.
After Reboot and with the problem the boot directory was now
drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire drwxr-xr-x 3 root root 4.0K Sep 30 2021 efi drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
Regards,
Adolf.
On 13/05/2022 12:09, Michael Tremer wrote:
Hello,
On 12 May 2022, at 21:10, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 12/05/2022 14:53, Michael Tremer wrote:
Hello,
On 12 May 2022, at 12:25, Adolf Belka adolf.belka@ipfire.org wrote:
Hi,
On 12/05/2022 11:13, Michael Tremer wrote:
Hello, Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while. So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved. This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started? The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
Okay. That is indeed quite bad then. Since there is no kernel in 168, is there a chance that we broke the update to 167?
I don't know but as far as I have been concerned CU167 has worked well on my vm. Is there something I should look for on my CU167 vm?
Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
Third time lucky isn’t good enough for me :)
Me neither :-)
I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
Have there been any modules loaded? If one loads, they all should load.
This is what I get with lsmod
Module Size Used by crct10dif_pclmul 16384 1 crc32_pclmul 16384 0 ghash_clmulni_intel 16384 0 serio_raw 20480 0 ohci_pci 20480 0 ata_generic 16384 0 pata_acpi 16384 0 video 57344 0
It looks like these modules could have come from the initramdisk image. At least that is working well.
On my running CU1678 vm lsmod gives the following
Module Size Used by tun 61440 2 nfnetlink_queue 28672 1 xt_NFQUEUE 16384 8 xt_MASQUERADE 20480 1 cfg80211 1036288 0 rfkill 32768 1 cfg80211 8021q 40960 0 garp 16384 1 8021q xt_set 16384 260 ip_set_hash_net 49152 258 ip_set 57344 2 xt_set,ip_set_hash_net xt_hashlimit 20480 2 xt_multiport 20480 4 xt_policy 16384 5 xt_TCPMSS 16384 1 xt_conntrack 16384 7 xt_comment 16384 18 ipt_REJECT 16384 1 nf_reject_ipv4 16384 1 ipt_REJECT xt_LOG 20480 26 xt_limit 16384 25 xt_mark 16384 27 xt_connmark 16384 2 nf_log_syslog 24576 26 iptable_raw 16384 0 iptable_mangle 16384 1 iptable_filter 16384 1 vfat 24576 1 fat 90112 1 vfat sch_cake 36864 4 intel_powerclamp 20480 0 psmouse 184320 0 pcspkr 16384 0 i2c_piix4 28672 0 e1000 163840 0 i2c_core 106496 2 psmouse,i2c_piix4 crct10dif_pclmul 16384 1 crc32_pclmul 16384 0 ata_generic 16384 0 pata_acpi 16384 0 ghash_clmulni_intel 16384 0 serio_raw 20480 0 ohci_pci 20480 0 video 57344 0
Yes, this looks more like it.
The last 8 entries are the same but a whole load of others are missing.
Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
-Michael
Regards, Adolf.
Tried modprobe e1000 but this came back with the following error modprobe: ERROR: could not insert 'e1000': Key was rejected by service
So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
Regards Adolf
On 11 May 2022, at 20:08, Adolf Belka adolf.belka@ipfire.org wrote:
Hi All,
On 11/05/2022 20:48, Jon Murphy wrote: > I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info: > *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* > Shouldn’t this be Core Update 168? That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
That seems to be a bug that should not be there. Did we recently update apache? -Michael
Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
I think I am going to raise a bug on this.
Regards, Adolf. > Jon >> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote: >> >> Hi All, >> >> On 11/05/2022 16:00, Adolf Belka wrote: >>> Hi Leo, >>> >>> On 11/05/2022 15:26, Leo Hofmann wrote: >>>> Hi Adolf, >>>> >>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code: >>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that. >>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 >>> Now I see it in the code. Thanks very much. >>>> >>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems! >>> Thanks for your help anyway. It's got rid of one of my questions. >>> >>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. >>> Will wait to hear other inputs on this problem. >>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again. >>> >> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones. >> >> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm. >> >> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/ >> >> Regards, >> >> Adolf. >> >>> Regards, >>> Adolf. >>> >>>> >>>> Best regards >>>> Leo >>>> >>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>>> Hi All, >>>>> >>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts. >>>>> >>>>> Normally my updates go without any real problems. This is the first time where this is not the case. >>>>> >>>>> >>>>> First try:- >>>>> >>>>> After running the update, which went quite quickly, the bottom of the screen had >>>>> >>>>> Core Update 167 Development Build: master/c22d834c >>>>> >>>>> I couldn't remember if this is what is expected or not for Testing releases. >>>>> >>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped. >>>>> >>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>> >>>>> Tried to access via the WUI and no access. Tried via SSH and no access. >>>>> >>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message >>>>> >>>>> starting >>>>> DUID 00:04:70:........ >>>>> red0: interface not found >>>>> >>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0. >>>>> >>>>> Looked in bootlog and this was filled with around 4000 lines all the same >>>>> >>>>> Loading of module with unavailable key is rejected >>>>> >>>>> >>>>> Second try:- >>>>> >>>>> This time running the update took over 10 mins. >>>>> >>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167. >>>>> >>>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>>> >>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time. >>>>> >>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28 >>>>> >>>>> Repository was set back to Stable. >>>>> >>>>> Changed again to Testing and it said again that there was an update from 167 to 168. >>>>> >>>>> Before running the pakfire directory had the following >>>>> >>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>> >>>>> After running it had >>>>> >>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >>>>> >>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen. >>>>> >>>>> Rebooted. >>>>> >>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c >>>>> >>>>> /opt/pakfire/db/core/mine has 168 in it >>>>> >>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168. >>>>> >>>>> >>>>> Third try:- >>>>> >>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>> >>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly. >>>>> >>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >>>>> >>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell. >>>>> >>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! >>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce >>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 >>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire >>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. >>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 >>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes >>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. >>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire >>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. >>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... >>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 >>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... >>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. >>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... >>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 >>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts... >>>>> >>>>> >>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. >>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce >>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... >>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. >>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire >>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. >>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... >>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing. >>>>> >>>>> Rebooted >>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>> Several repeats of the following two lines on the console screen:- >>>>> >>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) >>>>> Perhaps iptables or your kernel needs to be upgraded. >>>>> >>>>> >>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present. >>>>> >>>>> >>>>> I have given up at this point. >>>>> >>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know. >>>>> >>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this. >>>>> >>>>> Regards, >>>>> >>>>> Adolf.
Hi All,
I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
The vm shows the following
-bash-5.1$ cat /proc/mdstat Personalities : md127 : inactive sdb[1](S) 52428724 blocks super 1.0
unused devices: <none>
It looks like IPFire was only running on sda.
So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
The CU168 vm that is working has the same as the above setting.
The CU168 vm that is not working has the following setting
-bash-5.1$ cat /proc/mdstat Personalities : md127 : inactive sda[0](S) 52428724 blocks super 1.0
unused devices: <none>
So it looks like it has booted up on the disk that was not updated from the raid pair.
This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
It is probably dependent on which of the two disks actually get used to boot from.
So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
So raid systems need to have their raid arrays fully working before doing an upgrade.
At least the above is my interpretation of what I have found. What does everyone else think.
Regards,
Adolf.
On 13/05/2022 15:59, Adolf Belka wrote:
Hi Michael,
I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
However I did copy the boot directory info for these two cases and found an interesting effect.
For the problem vm here was the boot directory layout of the CU167 base machine
drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire drwxr-xr-x 3 root root 16K Jan 1 1970 efi drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
After upgrade and before reboot this had changed to
drwxr-xr-x 5 root root 4.0K May 13 15:40 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K May 3 04:02 config-5.15.35-ipfire drwxr-xr-x 3 root root 16K Jan 1 1970 efi drwxr-xr-x 6 root root 4.0K May 13 15:41 grub -rw------- 1 root root 26M May 13 15:41 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M May 3 04:02 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M May 3 04:02 vmlinuz-5.15.35-ipfire
Dates and times of some of the files/directories have changed as would be expected.
After Reboot and with the problem the boot directory was now
drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire drwxr-xr-x 3 root root 4.0K Sep 30 2021 efi drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
Regards,
Adolf.
On 13/05/2022 12:09, Michael Tremer wrote:
Hello,
On 12 May 2022, at 21:10, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 12/05/2022 14:53, Michael Tremer wrote:
Hello,
On 12 May 2022, at 12:25, Adolf Belka adolf.belka@ipfire.org wrote:
Hi,
On 12/05/2022 11:13, Michael Tremer wrote:
Hello, Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while. So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved. This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started? The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
Okay. That is indeed quite bad then. Since there is no kernel in 168, is there a chance that we broke the update to 167?
I don't know but as far as I have been concerned CU167 has worked well on my vm. Is there something I should look for on my CU167 vm?
Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
Third time lucky isn’t good enough for me :)
Me neither :-)
I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
Have there been any modules loaded? If one loads, they all should load.
This is what I get with lsmod
Module Size Used by crct10dif_pclmul 16384 1 crc32_pclmul 16384 0 ghash_clmulni_intel 16384 0 serio_raw 20480 0 ohci_pci 20480 0 ata_generic 16384 0 pata_acpi 16384 0 video 57344 0
It looks like these modules could have come from the initramdisk image. At least that is working well.
On my running CU1678 vm lsmod gives the following
Module Size Used by tun 61440 2 nfnetlink_queue 28672 1 xt_NFQUEUE 16384 8 xt_MASQUERADE 20480 1 cfg80211 1036288 0 rfkill 32768 1 cfg80211 8021q 40960 0 garp 16384 1 8021q xt_set 16384 260 ip_set_hash_net 49152 258 ip_set 57344 2 xt_set,ip_set_hash_net xt_hashlimit 20480 2 xt_multiport 20480 4 xt_policy 16384 5 xt_TCPMSS 16384 1 xt_conntrack 16384 7 xt_comment 16384 18 ipt_REJECT 16384 1 nf_reject_ipv4 16384 1 ipt_REJECT xt_LOG 20480 26 xt_limit 16384 25 xt_mark 16384 27 xt_connmark 16384 2 nf_log_syslog 24576 26 iptable_raw 16384 0 iptable_mangle 16384 1 iptable_filter 16384 1 vfat 24576 1 fat 90112 1 vfat sch_cake 36864 4 intel_powerclamp 20480 0 psmouse 184320 0 pcspkr 16384 0 i2c_piix4 28672 0 e1000 163840 0 i2c_core 106496 2 psmouse,i2c_piix4 crct10dif_pclmul 16384 1 crc32_pclmul 16384 0 ata_generic 16384 0 pata_acpi 16384 0 ghash_clmulni_intel 16384 0 serio_raw 20480 0 ohci_pci 20480 0 video 57344 0
Yes, this looks more like it.
The last 8 entries are the same but a whole load of others are missing.
Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
-Michael
Regards, Adolf.
Tried modprobe e1000 but this came back with the following error modprobe: ERROR: could not insert 'e1000': Key was rejected by service
So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
Regards Adolf
> On 11 May 2022, at 20:08, Adolf Belka adolf.belka@ipfire.org wrote: > > Hi All, > > On 11/05/2022 20:48, Jon Murphy wrote: >> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info: >> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* >> Shouldn’t this be Core Update 168? > That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases. > > I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm. > > I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box. > > I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box. > > Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box. That seems to be a bug that should not be there. Did we recently update apache? -Michael > > Don't know if this is related to the problem with the missing interfaces but if not then it is another problem. > > I think I am going to raise a bug on this. > > Regards, > Adolf. >> Jon >>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote: >>> >>> Hi All, >>> >>> On 11/05/2022 16:00, Adolf Belka wrote: >>>> Hi Leo, >>>> >>>> On 11/05/2022 15:26, Leo Hofmann wrote: >>>>> Hi Adolf, >>>>> >>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code: >>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that. >>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 >>>> Now I see it in the code. Thanks very much. >>>>> >>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems! >>>> Thanks for your help anyway. It's got rid of one of my questions. >>>> >>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. >>>> Will wait to hear other inputs on this problem. >>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again. >>>> >>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones. >>> >>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm. >>> >>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/ >>> >>> Regards, >>> >>> Adolf. >>> >>>> Regards, >>>> Adolf. >>>> >>>>> >>>>> Best regards >>>>> Leo >>>>> >>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>>>> Hi All, >>>>>> >>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts. >>>>>> >>>>>> Normally my updates go without any real problems. This is the first time where this is not the case. >>>>>> >>>>>> >>>>>> First try:- >>>>>> >>>>>> After running the update, which went quite quickly, the bottom of the screen had >>>>>> >>>>>> Core Update 167 Development Build: master/c22d834c >>>>>> >>>>>> I couldn't remember if this is what is expected or not for Testing releases. >>>>>> >>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped. >>>>>> >>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>> >>>>>> Tried to access via the WUI and no access. Tried via SSH and no access. >>>>>> >>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message >>>>>> >>>>>> starting >>>>>> DUID 00:04:70:........ >>>>>> red0: interface not found >>>>>> >>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0. >>>>>> >>>>>> Looked in bootlog and this was filled with around 4000 lines all the same >>>>>> >>>>>> Loading of module with unavailable key is rejected >>>>>> >>>>>> >>>>>> Second try:- >>>>>> >>>>>> This time running the update took over 10 mins. >>>>>> >>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167. >>>>>> >>>>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>>>> >>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time. >>>>>> >>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28 >>>>>> >>>>>> Repository was set back to Stable. >>>>>> >>>>>> Changed again to Testing and it said again that there was an update from 167 to 168. >>>>>> >>>>>> Before running the pakfire directory had the following >>>>>> >>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>> >>>>>> After running it had >>>>>> >>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >>>>>> >>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen. >>>>>> >>>>>> Rebooted. >>>>>> >>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c >>>>>> >>>>>> /opt/pakfire/db/core/mine has 168 in it >>>>>> >>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168. >>>>>> >>>>>> >>>>>> Third try:- >>>>>> >>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>> >>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly. >>>>>> >>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >>>>>> >>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell. >>>>>> >>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! >>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce >>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 >>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire >>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. >>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 >>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes >>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. >>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire >>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. >>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... >>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 >>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... >>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. >>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... >>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 >>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts... >>>>>> >>>>>> >>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. >>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce >>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... >>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. >>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire >>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. >>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... >>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing. >>>>>> >>>>>> Rebooted >>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>> Several repeats of the following two lines on the console screen:- >>>>>> >>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) >>>>>> Perhaps iptables or your kernel needs to be upgraded. >>>>>> >>>>>> >>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present. >>>>>> >>>>>> >>>>>> I have given up at this point. >>>>>> >>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know. >>>>>> >>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this. >>>>>> >>>>>> Regards, >>>>>> >>>>>> Adolf.
Hello,
Okay, this is really bad then if we are breaking RAID installations.
I tried to debug this, but VirtualBox is not my friend today.
I asked some bug reporter to provide me with the logs that I want over here:
https://bugzilla.ipfire.org/show_bug.cgi?id=12862
-Michael
On 13 May 2022, at 18:12, Adolf Belka adolf.belka@ipfire.org wrote:
Hi All,
I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
The vm shows the following
-bash-5.1$ cat /proc/mdstat Personalities : md127 : inactive sdb[1](S) 52428724 blocks super 1.0
unused devices: <none>
It looks like IPFire was only running on sda.
So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
The CU168 vm that is working has the same as the above setting.
The CU168 vm that is not working has the following setting
-bash-5.1$ cat /proc/mdstat Personalities : md127 : inactive sda[0](S) 52428724 blocks super 1.0
unused devices: <none>
So it looks like it has booted up on the disk that was not updated from the raid pair.
This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
It is probably dependent on which of the two disks actually get used to boot from.
So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
So raid systems need to have their raid arrays fully working before doing an upgrade.
At least the above is my interpretation of what I have found. What does everyone else think.
Regards,
Adolf.
On 13/05/2022 15:59, Adolf Belka wrote:
Hi Michael,
I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
However I did copy the boot directory info for these two cases and found an interesting effect.
For the problem vm here was the boot directory layout of the CU167 base machine
drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire drwxr-xr-x 3 root root 16K Jan 1 1970 efi drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
After upgrade and before reboot this had changed to
drwxr-xr-x 5 root root 4.0K May 13 15:40 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K May 3 04:02 config-5.15.35-ipfire drwxr-xr-x 3 root root 16K Jan 1 1970 efi drwxr-xr-x 6 root root 4.0K May 13 15:41 grub -rw------- 1 root root 26M May 13 15:41 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M May 3 04:02 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M May 3 04:02 vmlinuz-5.15.35-ipfire
Dates and times of some of the files/directories have changed as would be expected.
After Reboot and with the problem the boot directory was now
drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire drwxr-xr-x 3 root root 4.0K Sep 30 2021 efi drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
Regards,
Adolf.
On 13/05/2022 12:09, Michael Tremer wrote:
Hello,
On 12 May 2022, at 21:10, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 12/05/2022 14:53, Michael Tremer wrote:
Hello,
On 12 May 2022, at 12:25, Adolf Belka adolf.belka@ipfire.org wrote:
Hi,
On 12/05/2022 11:13, Michael Tremer wrote: > Hello, > Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while. > So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved. > This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started? > The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that. It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm. > No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case? No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
Okay. That is indeed quite bad then. Since there is no kernel in 168, is there a chance that we broke the update to 167?
I don't know but as far as I have been concerned CU167 has worked well on my vm. Is there something I should look for on my CU167 vm?
Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
Third time lucky isn’t good enough for me :)
Me neither :-)
I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
Have there been any modules loaded? If one loads, they all should load.
This is what I get with lsmod
Module Size Used by crct10dif_pclmul 16384 1 crc32_pclmul 16384 0 ghash_clmulni_intel 16384 0 serio_raw 20480 0 ohci_pci 20480 0 ata_generic 16384 0 pata_acpi 16384 0 video 57344 0
It looks like these modules could have come from the initramdisk image. At least that is working well.
On my running CU1678 vm lsmod gives the following
Module Size Used by tun 61440 2 nfnetlink_queue 28672 1 xt_NFQUEUE 16384 8 xt_MASQUERADE 20480 1 cfg80211 1036288 0 rfkill 32768 1 cfg80211 8021q 40960 0 garp 16384 1 8021q xt_set 16384 260 ip_set_hash_net 49152 258 ip_set 57344 2 xt_set,ip_set_hash_net xt_hashlimit 20480 2 xt_multiport 20480 4 xt_policy 16384 5 xt_TCPMSS 16384 1 xt_conntrack 16384 7 xt_comment 16384 18 ipt_REJECT 16384 1 nf_reject_ipv4 16384 1 ipt_REJECT xt_LOG 20480 26 xt_limit 16384 25 xt_mark 16384 27 xt_connmark 16384 2 nf_log_syslog 24576 26 iptable_raw 16384 0 iptable_mangle 16384 1 iptable_filter 16384 1 vfat 24576 1 fat 90112 1 vfat sch_cake 36864 4 intel_powerclamp 20480 0 psmouse 184320 0 pcspkr 16384 0 i2c_piix4 28672 0 e1000 163840 0 i2c_core 106496 2 psmouse,i2c_piix4 crct10dif_pclmul 16384 1 crc32_pclmul 16384 0 ata_generic 16384 0 pata_acpi 16384 0 ghash_clmulni_intel 16384 0 serio_raw 20480 0 ohci_pci 20480 0 video 57344 0
Yes, this looks more like it.
The last 8 entries are the same but a whole load of others are missing.
Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
-Michael
Regards, Adolf.
Tried modprobe e1000 but this came back with the following error modprobe: ERROR: could not insert 'e1000': Key was rejected by service
So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
Regards Adolf >> On 11 May 2022, at 20:08, Adolf Belka adolf.belka@ipfire.org wrote: >> >> Hi All, >> >> On 11/05/2022 20:48, Jon Murphy wrote: >>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info: >>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* >>> Shouldn’t this be Core Update 168? >> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases. >> >> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm. >> >> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box. >> >> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box. >> >> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box. > That seems to be a bug that should not be there. Did we recently update apache? > -Michael >> >> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem. >> >> I think I am going to raise a bug on this. >> >> Regards, >> Adolf. >>> Jon >>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote: >>>> >>>> Hi All, >>>> >>>> On 11/05/2022 16:00, Adolf Belka wrote: >>>>> Hi Leo, >>>>> >>>>> On 11/05/2022 15:26, Leo Hofmann wrote: >>>>>> Hi Adolf, >>>>>> >>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code: >>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that. >>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 >>>>> Now I see it in the code. Thanks very much. >>>>>> >>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems! >>>>> Thanks for your help anyway. It's got rid of one of my questions. >>>>> >>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. >>>>> Will wait to hear other inputs on this problem. >>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again. >>>>> >>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones. >>>> >>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm. >>>> >>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/ >>>> >>>> Regards, >>>> >>>> Adolf. >>>> >>>>> Regards, >>>>> Adolf. >>>>> >>>>>> >>>>>> Best regards >>>>>> Leo >>>>>> >>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>>>>> Hi All, >>>>>>> >>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts. >>>>>>> >>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case. >>>>>>> >>>>>>> >>>>>>> First try:- >>>>>>> >>>>>>> After running the update, which went quite quickly, the bottom of the screen had >>>>>>> >>>>>>> Core Update 167 Development Build: master/c22d834c >>>>>>> >>>>>>> I couldn't remember if this is what is expected or not for Testing releases. >>>>>>> >>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped. >>>>>>> >>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>> >>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access. >>>>>>> >>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message >>>>>>> >>>>>>> starting >>>>>>> DUID 00:04:70:........ >>>>>>> red0: interface not found >>>>>>> >>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0. >>>>>>> >>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same >>>>>>> >>>>>>> Loading of module with unavailable key is rejected >>>>>>> >>>>>>> >>>>>>> Second try:- >>>>>>> >>>>>>> This time running the update took over 10 mins. >>>>>>> >>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167. >>>>>>> >>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>>>>> >>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time. >>>>>>> >>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28 >>>>>>> >>>>>>> Repository was set back to Stable. >>>>>>> >>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168. >>>>>>> >>>>>>> Before running the pakfire directory had the following >>>>>>> >>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>> >>>>>>> After running it had >>>>>>> >>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >>>>>>> >>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen. >>>>>>> >>>>>>> Rebooted. >>>>>>> >>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c >>>>>>> >>>>>>> /opt/pakfire/db/core/mine has 168 in it >>>>>>> >>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168. >>>>>>> >>>>>>> >>>>>>> Third try:- >>>>>>> >>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>> >>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly. >>>>>>> >>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >>>>>>> >>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell. >>>>>>> >>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! >>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce >>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 >>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire >>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. >>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 >>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes >>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. >>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire >>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. >>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... >>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 >>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... >>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. >>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... >>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 >>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts... >>>>>>> >>>>>>> >>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. >>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce >>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... >>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. >>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire >>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. >>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... >>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing. >>>>>>> >>>>>>> Rebooted >>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>> Several repeats of the following two lines on the console screen:- >>>>>>> >>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) >>>>>>> Perhaps iptables or your kernel needs to be upgraded. >>>>>>> >>>>>>> >>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present. >>>>>>> >>>>>>> >>>>>>> I have given up at this point. >>>>>>> >>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know. >>>>>>> >>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this. >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Adolf.
Hi Michael,
On 13/05/2022 22:37, Michael Tremer wrote:
Hello,
Okay, this is really bad then if we are breaking RAID installations.
I tried to debug this, but VirtualBox is not my friend today.
I asked some bug reporter to provide me with the logs that I want over here:
After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
Thanks very much for all the help.
Regards, Adolf.
-Michael
On 13 May 2022, at 18:12, Adolf Belka adolf.belka@ipfire.org wrote:
Hi All,
I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
The vm shows the following
-bash-5.1$ cat /proc/mdstat Personalities : md127 : inactive sdb[1](S) 52428724 blocks super 1.0
unused devices: <none>
It looks like IPFire was only running on sda.
So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
The CU168 vm that is working has the same as the above setting.
The CU168 vm that is not working has the following setting
-bash-5.1$ cat /proc/mdstat Personalities : md127 : inactive sda[0](S) 52428724 blocks super 1.0
unused devices: <none>
So it looks like it has booted up on the disk that was not updated from the raid pair.
This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
It is probably dependent on which of the two disks actually get used to boot from.
So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
So raid systems need to have their raid arrays fully working before doing an upgrade.
At least the above is my interpretation of what I have found. What does everyone else think.
Regards,
Adolf.
On 13/05/2022 15:59, Adolf Belka wrote:
Hi Michael,
I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
However I did copy the boot directory info for these two cases and found an interesting effect.
For the problem vm here was the boot directory layout of the CU167 base machine
drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire drwxr-xr-x 3 root root 16K Jan 1 1970 efi drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
After upgrade and before reboot this had changed to
drwxr-xr-x 5 root root 4.0K May 13 15:40 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K May 3 04:02 config-5.15.35-ipfire drwxr-xr-x 3 root root 16K Jan 1 1970 efi drwxr-xr-x 6 root root 4.0K May 13 15:41 grub -rw------- 1 root root 26M May 13 15:41 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M May 3 04:02 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M May 3 04:02 vmlinuz-5.15.35-ipfire
Dates and times of some of the files/directories have changed as would be expected.
After Reboot and with the problem the boot directory was now
drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire drwxr-xr-x 3 root root 4.0K Sep 30 2021 efi drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
Regards,
Adolf.
On 13/05/2022 12:09, Michael Tremer wrote:
Hello,
On 12 May 2022, at 21:10, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 12/05/2022 14:53, Michael Tremer wrote:
Hello, > On 12 May 2022, at 12:25, Adolf Belka adolf.belka@ipfire.org wrote: > > Hi, > > On 12/05/2022 11:13, Michael Tremer wrote: >> Hello, >> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while. >> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved. >> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started? >> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that. > It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm. >> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case? > No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces. Okay. That is indeed quite bad then. Since there is no kernel in 168, is there a chance that we broke the update to 167?
I don't know but as far as I have been concerned CU167 has worked well on my vm. Is there something I should look for on my CU167 vm?
> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface. Third time lucky isn’t good enough for me :)
Me neither :-)
> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces. > > I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers Have there been any modules loaded? If one loads, they all should load.
This is what I get with lsmod
Module Size Used by crct10dif_pclmul 16384 1 crc32_pclmul 16384 0 ghash_clmulni_intel 16384 0 serio_raw 20480 0 ohci_pci 20480 0 ata_generic 16384 0 pata_acpi 16384 0 video 57344 0
It looks like these modules could have come from the initramdisk image. At least that is working well.
On my running CU1678 vm lsmod gives the following
Module Size Used by tun 61440 2 nfnetlink_queue 28672 1 xt_NFQUEUE 16384 8 xt_MASQUERADE 20480 1 cfg80211 1036288 0 rfkill 32768 1 cfg80211 8021q 40960 0 garp 16384 1 8021q xt_set 16384 260 ip_set_hash_net 49152 258 ip_set 57344 2 xt_set,ip_set_hash_net xt_hashlimit 20480 2 xt_multiport 20480 4 xt_policy 16384 5 xt_TCPMSS 16384 1 xt_conntrack 16384 7 xt_comment 16384 18 ipt_REJECT 16384 1 nf_reject_ipv4 16384 1 ipt_REJECT xt_LOG 20480 26 xt_limit 16384 25 xt_mark 16384 27 xt_connmark 16384 2 nf_log_syslog 24576 26 iptable_raw 16384 0 iptable_mangle 16384 1 iptable_filter 16384 1 vfat 24576 1 fat 90112 1 vfat sch_cake 36864 4 intel_powerclamp 20480 0 psmouse 184320 0 pcspkr 16384 0 i2c_piix4 28672 0 e1000 163840 0 i2c_core 106496 2 psmouse,i2c_piix4 crct10dif_pclmul 16384 1 crc32_pclmul 16384 0 ata_generic 16384 0 pata_acpi 16384 0 ghash_clmulni_intel 16384 0 serio_raw 20480 0 ohci_pci 20480 0 video 57344 0
Yes, this looks more like it.
The last 8 entries are the same but a whole load of others are missing.
Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
-Michael
Regards, Adolf.
> Tried modprobe e1000 but this came back with the following error > modprobe: ERROR: could not insert 'e1000': Key was rejected by service > > So out of 7 or 8 reboots the vm booted with the network drivers loaded once. > > Regards > Adolf >>> On 11 May 2022, at 20:08, Adolf Belka adolf.belka@ipfire.org wrote: >>> >>> Hi All, >>> >>> On 11/05/2022 20:48, Jon Murphy wrote: >>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info: >>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* >>>> Shouldn’t this be Core Update 168? >>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases. >>> >>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm. >>> >>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box. >>> >>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box. >>> >>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box. >> That seems to be a bug that should not be there. Did we recently update apache? >> -Michael >>> >>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem. >>> >>> I think I am going to raise a bug on this. >>> >>> Regards, >>> Adolf. >>>> Jon >>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote: >>>>> >>>>> Hi All, >>>>> >>>>> On 11/05/2022 16:00, Adolf Belka wrote: >>>>>> Hi Leo, >>>>>> >>>>>> On 11/05/2022 15:26, Leo Hofmann wrote: >>>>>>> Hi Adolf, >>>>>>> >>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code: >>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that. >>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 >>>>>> Now I see it in the code. Thanks very much. >>>>>>> >>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems! >>>>>> Thanks for your help anyway. It's got rid of one of my questions. >>>>>> >>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. >>>>>> Will wait to hear other inputs on this problem. >>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again. >>>>>> >>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones. >>>>> >>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm. >>>>> >>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/ >>>>> >>>>> Regards, >>>>> >>>>> Adolf. >>>>> >>>>>> Regards, >>>>>> Adolf. >>>>>> >>>>>>> >>>>>>> Best regards >>>>>>> Leo >>>>>>> >>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>>>>>> Hi All, >>>>>>>> >>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts. >>>>>>>> >>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case. >>>>>>>> >>>>>>>> >>>>>>>> First try:- >>>>>>>> >>>>>>>> After running the update, which went quite quickly, the bottom of the screen had >>>>>>>> >>>>>>>> Core Update 167 Development Build: master/c22d834c >>>>>>>> >>>>>>>> I couldn't remember if this is what is expected or not for Testing releases. >>>>>>>> >>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped. >>>>>>>> >>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>> >>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access. >>>>>>>> >>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message >>>>>>>> >>>>>>>> starting >>>>>>>> DUID 00:04:70:........ >>>>>>>> red0: interface not found >>>>>>>> >>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0. >>>>>>>> >>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same >>>>>>>> >>>>>>>> Loading of module with unavailable key is rejected >>>>>>>> >>>>>>>> >>>>>>>> Second try:- >>>>>>>> >>>>>>>> This time running the update took over 10 mins. >>>>>>>> >>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167. >>>>>>>> >>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>>>>>> >>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time. >>>>>>>> >>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28 >>>>>>>> >>>>>>>> Repository was set back to Stable. >>>>>>>> >>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168. >>>>>>>> >>>>>>>> Before running the pakfire directory had the following >>>>>>>> >>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>> >>>>>>>> After running it had >>>>>>>> >>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >>>>>>>> >>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen. >>>>>>>> >>>>>>>> Rebooted. >>>>>>>> >>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c >>>>>>>> >>>>>>>> /opt/pakfire/db/core/mine has 168 in it >>>>>>>> >>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168. >>>>>>>> >>>>>>>> >>>>>>>> Third try:- >>>>>>>> >>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>> >>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly. >>>>>>>> >>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >>>>>>>> >>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell. >>>>>>>> >>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! >>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce >>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 >>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire >>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. >>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 >>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes >>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. >>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire >>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. >>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... >>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 >>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... >>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. >>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... >>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 >>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts... >>>>>>>> >>>>>>>> >>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. >>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce >>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... >>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. >>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire >>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. >>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... >>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing. >>>>>>>> >>>>>>>> Rebooted >>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>> Several repeats of the following two lines on the console screen:- >>>>>>>> >>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) >>>>>>>> Perhaps iptables or your kernel needs to be upgraded. >>>>>>>> >>>>>>>> >>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present. >>>>>>>> >>>>>>>> >>>>>>>> I have given up at this point. >>>>>>>> >>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know. >>>>>>>> >>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this. >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Adolf.
Hi All,
Virtually everything I have tested in CU168 looks to be working as expected.
The only problem I have found to-date is the Rules update for Suricata in the Intrusion Prevention System.
The old daily or weekly option is gone and the update should now occur twice per day but that entry is not in the fcrontab in CU168 Testing so no updates are occurring for the rulesets unless you manually force them when the update occurs with no problem.
Regards,
Adolf
On 17/05/2022 10:09, Adolf Belka wrote:
Hi Michael,
On 13/05/2022 22:37, Michael Tremer wrote:
Hello,
Okay, this is really bad then if we are breaking RAID installations.
I tried to debug this, but VirtualBox is not my friend today.
I asked some bug reporter to provide me with the logs that I want over here:
After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
Thanks very much for all the help.
Regards, Adolf.
-Michael
On 13 May 2022, at 18:12, Adolf Belka adolf.belka@ipfire.org wrote:
Hi All,
I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
The vm shows the following
-bash-5.1$ cat /proc/mdstat Personalities : md127 : inactive sdb[1](S) 52428724 blocks super 1.0
unused devices: <none>
It looks like IPFire was only running on sda.
So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
The CU168 vm that is working has the same as the above setting.
The CU168 vm that is not working has the following setting
-bash-5.1$ cat /proc/mdstat Personalities : md127 : inactive sda[0](S) 52428724 blocks super 1.0
unused devices: <none>
So it looks like it has booted up on the disk that was not updated from the raid pair.
This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
It is probably dependent on which of the two disks actually get used to boot from.
So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
So raid systems need to have their raid arrays fully working before doing an upgrade.
At least the above is my interpretation of what I have found. What does everyone else think.
Regards,
Adolf.
On 13/05/2022 15:59, Adolf Belka wrote:
Hi Michael,
I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
However I did copy the boot directory info for these two cases and found an interesting effect.
For the problem vm here was the boot directory layout of the CU167 base machine
drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire drwxr-xr-x 3 root root 16K Jan 1 1970 efi drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
After upgrade and before reboot this had changed to
drwxr-xr-x 5 root root 4.0K May 13 15:40 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K May 3 04:02 config-5.15.35-ipfire drwxr-xr-x 3 root root 16K Jan 1 1970 efi drwxr-xr-x 6 root root 4.0K May 13 15:41 grub -rw------- 1 root root 26M May 13 15:41 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M May 3 04:02 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M May 3 04:02 vmlinuz-5.15.35-ipfire
Dates and times of some of the files/directories have changed as would be expected.
After Reboot and with the problem the boot directory was now
drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire drwxr-xr-x 3 root root 4.0K Sep 30 2021 efi drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
Regards,
Adolf.
On 13/05/2022 12:09, Michael Tremer wrote:
Hello,
On 12 May 2022, at 21:10, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 12/05/2022 14:53, Michael Tremer wrote: > Hello, >> On 12 May 2022, at 12:25, Adolf Belka adolf.belka@ipfire.org wrote: >> >> Hi, >> >> On 12/05/2022 11:13, Michael Tremer wrote: >>> Hello, >>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while. >>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved. >>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started? >>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that. >> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm. >>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case? >> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces. > Okay. That is indeed quite bad then. > Since there is no kernel in 168, is there a chance that we broke the update to 167? I don't know but as far as I have been concerned CU167 has worked well on my vm. Is there something I should look for on my CU167 vm? >> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface. > Third time lucky isn’t good enough for me :) Me neither :-) >> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces. >> >> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers > Have there been any modules loaded? If one loads, they all should load. This is what I get with lsmod
Module Size Used by crct10dif_pclmul 16384 1 crc32_pclmul 16384 0 ghash_clmulni_intel 16384 0 serio_raw 20480 0 ohci_pci 20480 0 ata_generic 16384 0 pata_acpi 16384 0 video 57344 0
It looks like these modules could have come from the initramdisk image. At least that is working well.
On my running CU1678 vm lsmod gives the following
Module Size Used by tun 61440 2 nfnetlink_queue 28672 1 xt_NFQUEUE 16384 8 xt_MASQUERADE 20480 1 cfg80211 1036288 0 rfkill 32768 1 cfg80211 8021q 40960 0 garp 16384 1 8021q xt_set 16384 260 ip_set_hash_net 49152 258 ip_set 57344 2 xt_set,ip_set_hash_net xt_hashlimit 20480 2 xt_multiport 20480 4 xt_policy 16384 5 xt_TCPMSS 16384 1 xt_conntrack 16384 7 xt_comment 16384 18 ipt_REJECT 16384 1 nf_reject_ipv4 16384 1 ipt_REJECT xt_LOG 20480 26 xt_limit 16384 25 xt_mark 16384 27 xt_connmark 16384 2 nf_log_syslog 24576 26 iptable_raw 16384 0 iptable_mangle 16384 1 iptable_filter 16384 1 vfat 24576 1 fat 90112 1 vfat sch_cake 36864 4 intel_powerclamp 20480 0 psmouse 184320 0 pcspkr 16384 0 i2c_piix4 28672 0 e1000 163840 0 i2c_core 106496 2 psmouse,i2c_piix4 crct10dif_pclmul 16384 1 crc32_pclmul 16384 0 ata_generic 16384 0 pata_acpi 16384 0 ghash_clmulni_intel 16384 0 serio_raw 20480 0 ohci_pci 20480 0 video 57344 0
Yes, this looks more like it.
The last 8 entries are the same but a whole load of others are missing.
Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
-Michael
Regards, Adolf. >> Tried modprobe e1000 but this came back with the following error >> modprobe: ERROR: could not insert 'e1000': Key was rejected by service >> >> So out of 7 or 8 reboots the vm booted with the network drivers loaded once. >> >> Regards >> Adolf >>>> On 11 May 2022, at 20:08, Adolf Belka adolf.belka@ipfire.org wrote: >>>> >>>> Hi All, >>>> >>>> On 11/05/2022 20:48, Jon Murphy wrote: >>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info: >>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* >>>>> Shouldn’t this be Core Update 168? >>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases. >>>> >>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm. >>>> >>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box. >>>> >>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box. >>>> >>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box. >>> That seems to be a bug that should not be there. Did we recently update apache? >>> -Michael >>>> >>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem. >>>> >>>> I think I am going to raise a bug on this. >>>> >>>> Regards, >>>> Adolf. >>>>> Jon >>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote: >>>>>> >>>>>> Hi All, >>>>>> >>>>>> On 11/05/2022 16:00, Adolf Belka wrote: >>>>>>> Hi Leo, >>>>>>> >>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote: >>>>>>>> Hi Adolf, >>>>>>>> >>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code: >>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that. >>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 >>>>>>> Now I see it in the code. Thanks very much. >>>>>>>> >>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems! >>>>>>> Thanks for your help anyway. It's got rid of one of my questions. >>>>>>> >>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. >>>>>>> Will wait to hear other inputs on this problem. >>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again. >>>>>>> >>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones. >>>>>> >>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm. >>>>>> >>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/ >>>>>> >>>>>> Regards, >>>>>> >>>>>> Adolf. >>>>>> >>>>>>> Regards, >>>>>>> Adolf. >>>>>>> >>>>>>>> >>>>>>>> Best regards >>>>>>>> Leo >>>>>>>> >>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts. >>>>>>>>> >>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case. >>>>>>>>> >>>>>>>>> >>>>>>>>> First try:- >>>>>>>>> >>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had >>>>>>>>> >>>>>>>>> Core Update 167 Development Build: master/c22d834c >>>>>>>>> >>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases. >>>>>>>>> >>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped. >>>>>>>>> >>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>> >>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access. >>>>>>>>> >>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message >>>>>>>>> >>>>>>>>> starting >>>>>>>>> DUID 00:04:70:........ >>>>>>>>> red0: interface not found >>>>>>>>> >>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0. >>>>>>>>> >>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same >>>>>>>>> >>>>>>>>> Loading of module with unavailable key is rejected >>>>>>>>> >>>>>>>>> >>>>>>>>> Second try:- >>>>>>>>> >>>>>>>>> This time running the update took over 10 mins. >>>>>>>>> >>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167. >>>>>>>>> >>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>>>>>>> >>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time. >>>>>>>>> >>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28 >>>>>>>>> >>>>>>>>> Repository was set back to Stable. >>>>>>>>> >>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168. >>>>>>>>> >>>>>>>>> Before running the pakfire directory had the following >>>>>>>>> >>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>> >>>>>>>>> After running it had >>>>>>>>> >>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >>>>>>>>> >>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen. >>>>>>>>> >>>>>>>>> Rebooted. >>>>>>>>> >>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c >>>>>>>>> >>>>>>>>> /opt/pakfire/db/core/mine has 168 in it >>>>>>>>> >>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168. >>>>>>>>> >>>>>>>>> >>>>>>>>> Third try:- >>>>>>>>> >>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>> >>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly. >>>>>>>>> >>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >>>>>>>>> >>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell. >>>>>>>>> >>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! >>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce >>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 >>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire >>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. >>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 >>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes >>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. >>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire >>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. >>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... >>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 >>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... >>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. >>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... >>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 >>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts... >>>>>>>>> >>>>>>>>> >>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. >>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce >>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... >>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. >>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire >>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. >>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... >>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing. >>>>>>>>> >>>>>>>>> Rebooted >>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>> Several repeats of the following two lines on the console screen:- >>>>>>>>> >>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) >>>>>>>>> Perhaps iptables or your kernel needs to be upgraded. >>>>>>>>> >>>>>>>>> >>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present. >>>>>>>>> >>>>>>>>> >>>>>>>>> I have given up at this point. >>>>>>>>> >>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know. >>>>>>>>> >>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Adolf.
Hello Adolf,
thank you for all the extensive testing and reporting back.
Ah, indeed, the newly introduced Cron job is stil missing due to silly me confusing the fcrontab binary and an actual fcrontab. I think the most straight-forward way to fix this is to just ship the entire crontab file, and force fcrontab to recreate the - yes - fcrontab based on its content.
@Michael: How does one ship a single file with a Core Update that is not covered by any rootfile?
Thanks, and best regards, Peter Müller
Hi All,
Virtually everything I have tested in CU168 looks to be working as expected.
The only problem I have found to-date is the Rules update for Suricata in the Intrusion Prevention System.
The old daily or weekly option is gone and the update should now occur twice per day but that entry is not in the fcrontab in CU168 Testing so no updates are occurring for the rulesets unless you manually force them when the update occurs with no problem.
Regards,
Adolf
On 17/05/2022 10:09, Adolf Belka wrote:
Hi Michael,
On 13/05/2022 22:37, Michael Tremer wrote:
Hello,
Okay, this is really bad then if we are breaking RAID installations.
I tried to debug this, but VirtualBox is not my friend today.
I asked some bug reporter to provide me with the logs that I want over here:
After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
Thanks very much for all the help.
Regards, Adolf.
-Michael
On 13 May 2022, at 18:12, Adolf Belka adolf.belka@ipfire.org wrote:
Hi All,
I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
The vm shows the following
-bash-5.1$ cat /proc/mdstat Personalities : md127 : inactive sdb[1](S) 52428724 blocks super 1.0
unused devices: <none>
It looks like IPFire was only running on sda.
So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
The CU168 vm that is working has the same as the above setting.
The CU168 vm that is not working has the following setting
-bash-5.1$ cat /proc/mdstat Personalities : md127 : inactive sda[0](S) 52428724 blocks super 1.0
unused devices: <none>
So it looks like it has booted up on the disk that was not updated from the raid pair.
This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
It is probably dependent on which of the two disks actually get used to boot from.
So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
So raid systems need to have their raid arrays fully working before doing an upgrade.
At least the above is my interpretation of what I have found. What does everyone else think.
Regards,
Adolf.
On 13/05/2022 15:59, Adolf Belka wrote:
Hi Michael,
I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
However I did copy the boot directory info for these two cases and found an interesting effect.
For the problem vm here was the boot directory layout of the CU167 base machine
drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire drwxr-xr-x 3 root root 16K Jan 1 1970 efi drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
After upgrade and before reboot this had changed to
drwxr-xr-x 5 root root 4.0K May 13 15:40 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K May 3 04:02 config-5.15.35-ipfire drwxr-xr-x 3 root root 16K Jan 1 1970 efi drwxr-xr-x 6 root root 4.0K May 13 15:41 grub -rw------- 1 root root 26M May 13 15:41 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M May 3 04:02 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M May 3 04:02 vmlinuz-5.15.35-ipfire
Dates and times of some of the files/directories have changed as would be expected.
After Reboot and with the problem the boot directory was now
drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire drwxr-xr-x 3 root root 4.0K Sep 30 2021 efi drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
Regards,
Adolf.
On 13/05/2022 12:09, Michael Tremer wrote:
Hello,
> On 12 May 2022, at 21:10, Adolf Belka adolf.belka@ipfire.org wrote: > > Hi Michael, > > On 12/05/2022 14:53, Michael Tremer wrote: >> Hello, >>> On 12 May 2022, at 12:25, Adolf Belka adolf.belka@ipfire.org wrote: >>> >>> Hi, >>> >>> On 12/05/2022 11:13, Michael Tremer wrote: >>>> Hello, >>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while. >>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved. >>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started? >>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that. >>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm. >>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case? >>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces. >> Okay. That is indeed quite bad then. >> Since there is no kernel in 168, is there a chance that we broke the update to 167? > I don't know but as far as I have been concerned CU167 has worked well on my vm. > Is there something I should look for on my CU167 vm? >>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface. >> Third time lucky isn’t good enough for me :) > Me neither :-) >>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces. >>> >>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers >> Have there been any modules loaded? If one loads, they all should load. > This is what I get with lsmod > > Module Size Used by > crct10dif_pclmul 16384 1 > crc32_pclmul 16384 0 > ghash_clmulni_intel 16384 0 > serio_raw 20480 0 > ohci_pci 20480 0 > ata_generic 16384 0 > pata_acpi 16384 0 > video 57344 0
It looks like these modules could have come from the initramdisk image. At least that is working well.
> On my running CU1678 vm lsmod gives the following > > Module Size Used by > tun 61440 2 > nfnetlink_queue 28672 1 > xt_NFQUEUE 16384 8 > xt_MASQUERADE 20480 1 > cfg80211 1036288 0 > rfkill 32768 1 cfg80211 > 8021q 40960 0 > garp 16384 1 8021q > xt_set 16384 260 > ip_set_hash_net 49152 258 > ip_set 57344 2 xt_set,ip_set_hash_net > xt_hashlimit 20480 2 > xt_multiport 20480 4 > xt_policy 16384 5 > xt_TCPMSS 16384 1 > xt_conntrack 16384 7 > xt_comment 16384 18 > ipt_REJECT 16384 1 > nf_reject_ipv4 16384 1 ipt_REJECT > xt_LOG 20480 26 > xt_limit 16384 25 > xt_mark 16384 27 > xt_connmark 16384 2 > nf_log_syslog 24576 26 > iptable_raw 16384 0 > iptable_mangle 16384 1 > iptable_filter 16384 1 > vfat 24576 1 > fat 90112 1 vfat > sch_cake 36864 4 > intel_powerclamp 20480 0 > psmouse 184320 0 > pcspkr 16384 0 > i2c_piix4 28672 0 > e1000 163840 0 > i2c_core 106496 2 psmouse,i2c_piix4 > crct10dif_pclmul 16384 1 > crc32_pclmul 16384 0 > ata_generic 16384 0 > pata_acpi 16384 0 > ghash_clmulni_intel 16384 0 > serio_raw 20480 0 > ohci_pci 20480 0 > video 57344 0
Yes, this looks more like it.
> The last 8 entries are the same but a whole load of others are missing.
Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
-Michael
> > Regards, > Adolf. >>> Tried modprobe e1000 but this came back with the following error >>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service >>> >>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once. >>> >>> Regards >>> Adolf >>>>> On 11 May 2022, at 20:08, Adolf Belka adolf.belka@ipfire.org wrote: >>>>> >>>>> Hi All, >>>>> >>>>> On 11/05/2022 20:48, Jon Murphy wrote: >>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info: >>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* >>>>>> Shouldn’t this be Core Update 168? >>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases. >>>>> >>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm. >>>>> >>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box. >>>>> >>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box. >>>>> >>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box. >>>> That seems to be a bug that should not be there. Did we recently update apache? >>>> -Michael >>>>> >>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem. >>>>> >>>>> I think I am going to raise a bug on this. >>>>> >>>>> Regards, >>>>> Adolf. >>>>>> Jon >>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote: >>>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> On 11/05/2022 16:00, Adolf Belka wrote: >>>>>>>> Hi Leo, >>>>>>>> >>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote: >>>>>>>>> Hi Adolf, >>>>>>>>> >>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code: >>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that. >>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 >>>>>>>> Now I see it in the code. Thanks very much. >>>>>>>>> >>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems! >>>>>>>> Thanks for your help anyway. It's got rid of one of my questions. >>>>>>>> >>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. >>>>>>>> Will wait to hear other inputs on this problem. >>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again. >>>>>>>> >>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones. >>>>>>> >>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm. >>>>>>> >>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/ >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Adolf. >>>>>>> >>>>>>>> Regards, >>>>>>>> Adolf. >>>>>>>> >>>>>>>>> >>>>>>>>> Best regards >>>>>>>>> Leo >>>>>>>>> >>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts. >>>>>>>>>> >>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> First try:- >>>>>>>>>> >>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had >>>>>>>>>> >>>>>>>>>> Core Update 167 Development Build: master/c22d834c >>>>>>>>>> >>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases. >>>>>>>>>> >>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped. >>>>>>>>>> >>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>>> >>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access. >>>>>>>>>> >>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message >>>>>>>>>> >>>>>>>>>> starting >>>>>>>>>> DUID 00:04:70:........ >>>>>>>>>> red0: interface not found >>>>>>>>>> >>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0. >>>>>>>>>> >>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same >>>>>>>>>> >>>>>>>>>> Loading of module with unavailable key is rejected >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Second try:- >>>>>>>>>> >>>>>>>>>> This time running the update took over 10 mins. >>>>>>>>>> >>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167. >>>>>>>>>> >>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>>>>>>>> >>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time. >>>>>>>>>> >>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28 >>>>>>>>>> >>>>>>>>>> Repository was set back to Stable. >>>>>>>>>> >>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168. >>>>>>>>>> >>>>>>>>>> Before running the pakfire directory had the following >>>>>>>>>> >>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>>> >>>>>>>>>> After running it had >>>>>>>>>> >>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >>>>>>>>>> >>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen. >>>>>>>>>> >>>>>>>>>> Rebooted. >>>>>>>>>> >>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c >>>>>>>>>> >>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it >>>>>>>>>> >>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Third try:- >>>>>>>>>> >>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>>> >>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly. >>>>>>>>>> >>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >>>>>>>>>> >>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell. >>>>>>>>>> >>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! >>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce >>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 >>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire >>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. >>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 >>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes >>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. >>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire >>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. >>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... >>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 >>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... >>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. >>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... >>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 >>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts... >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. >>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce >>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... >>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. >>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire >>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. >>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... >>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing. >>>>>>>>>> >>>>>>>>>> Rebooted >>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>>> Several repeats of the following two lines on the console screen:- >>>>>>>>>> >>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) >>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I have given up at this point. >>>>>>>>>> >>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know. >>>>>>>>>> >>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Adolf.
Hello Michael,
as promised earlier today:
Hello Adolf,
thank you for all the extensive testing and reporting back.
Ah, indeed, the newly introduced Cron job is stil missing due to silly me confusing the fcrontab binary and an actual fcrontab. I think the most straight-forward way to fix this is to just ship the entire crontab file, and force fcrontab to recreate the
- yes - fcrontab based on its content.
@Michael: How does one ship a single file with a Core Update that is not covered by any rootfile?
- ping - :-)
Thanks, and best regards, Peter Müller
Thanks, and best regards, Peter Müller
Hi All,
Virtually everything I have tested in CU168 looks to be working as expected.
The only problem I have found to-date is the Rules update for Suricata in the Intrusion Prevention System.
The old daily or weekly option is gone and the update should now occur twice per day but that entry is not in the fcrontab in CU168 Testing so no updates are occurring for the rulesets unless you manually force them when the update occurs with no problem.
Regards,
Adolf
On 17/05/2022 10:09, Adolf Belka wrote:
Hi Michael,
On 13/05/2022 22:37, Michael Tremer wrote:
Hello,
Okay, this is really bad then if we are breaking RAID installations.
I tried to debug this, but VirtualBox is not my friend today.
I asked some bug reporter to provide me with the logs that I want over here:
After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
Thanks very much for all the help.
Regards, Adolf.
-Michael
On 13 May 2022, at 18:12, Adolf Belka adolf.belka@ipfire.org wrote:
Hi All,
I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
The vm shows the following
-bash-5.1$ cat /proc/mdstat Personalities : md127 : inactive sdb[1](S) 52428724 blocks super 1.0
unused devices: <none>
It looks like IPFire was only running on sda.
So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
The CU168 vm that is working has the same as the above setting.
The CU168 vm that is not working has the following setting
-bash-5.1$ cat /proc/mdstat Personalities : md127 : inactive sda[0](S) 52428724 blocks super 1.0
unused devices: <none>
So it looks like it has booted up on the disk that was not updated from the raid pair.
This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
It is probably dependent on which of the two disks actually get used to boot from.
So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
So raid systems need to have their raid arrays fully working before doing an upgrade.
At least the above is my interpretation of what I have found. What does everyone else think.
Regards,
Adolf.
On 13/05/2022 15:59, Adolf Belka wrote:
Hi Michael,
I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
However I did copy the boot directory info for these two cases and found an interesting effect.
For the problem vm here was the boot directory layout of the CU167 base machine
drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire drwxr-xr-x 3 root root 16K Jan 1 1970 efi drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
After upgrade and before reboot this had changed to
drwxr-xr-x 5 root root 4.0K May 13 15:40 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K May 3 04:02 config-5.15.35-ipfire drwxr-xr-x 3 root root 16K Jan 1 1970 efi drwxr-xr-x 6 root root 4.0K May 13 15:41 grub -rw------- 1 root root 26M May 13 15:41 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M May 3 04:02 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M May 3 04:02 vmlinuz-5.15.35-ipfire
Dates and times of some of the files/directories have changed as would be expected.
After Reboot and with the problem the boot directory was now
drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire drwxr-xr-x 3 root root 4.0K Sep 30 2021 efi drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
Regards,
Adolf.
On 13/05/2022 12:09, Michael Tremer wrote: > Hello, > >> On 12 May 2022, at 21:10, Adolf Belka adolf.belka@ipfire.org wrote: >> >> Hi Michael, >> >> On 12/05/2022 14:53, Michael Tremer wrote: >>> Hello, >>>> On 12 May 2022, at 12:25, Adolf Belka adolf.belka@ipfire.org wrote: >>>> >>>> Hi, >>>> >>>> On 12/05/2022 11:13, Michael Tremer wrote: >>>>> Hello, >>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while. >>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved. >>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started? >>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that. >>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm. >>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case? >>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces. >>> Okay. That is indeed quite bad then. >>> Since there is no kernel in 168, is there a chance that we broke the update to 167? >> I don't know but as far as I have been concerned CU167 has worked well on my vm. >> Is there something I should look for on my CU167 vm? >>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface. >>> Third time lucky isn’t good enough for me :) >> Me neither :-) >>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces. >>>> >>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers >>> Have there been any modules loaded? If one loads, they all should load. >> This is what I get with lsmod >> >> Module Size Used by >> crct10dif_pclmul 16384 1 >> crc32_pclmul 16384 0 >> ghash_clmulni_intel 16384 0 >> serio_raw 20480 0 >> ohci_pci 20480 0 >> ata_generic 16384 0 >> pata_acpi 16384 0 >> video 57344 0 > > It looks like these modules could have come from the initramdisk image. At least that is working well. > >> On my running CU1678 vm lsmod gives the following >> >> Module Size Used by >> tun 61440 2 >> nfnetlink_queue 28672 1 >> xt_NFQUEUE 16384 8 >> xt_MASQUERADE 20480 1 >> cfg80211 1036288 0 >> rfkill 32768 1 cfg80211 >> 8021q 40960 0 >> garp 16384 1 8021q >> xt_set 16384 260 >> ip_set_hash_net 49152 258 >> ip_set 57344 2 xt_set,ip_set_hash_net >> xt_hashlimit 20480 2 >> xt_multiport 20480 4 >> xt_policy 16384 5 >> xt_TCPMSS 16384 1 >> xt_conntrack 16384 7 >> xt_comment 16384 18 >> ipt_REJECT 16384 1 >> nf_reject_ipv4 16384 1 ipt_REJECT >> xt_LOG 20480 26 >> xt_limit 16384 25 >> xt_mark 16384 27 >> xt_connmark 16384 2 >> nf_log_syslog 24576 26 >> iptable_raw 16384 0 >> iptable_mangle 16384 1 >> iptable_filter 16384 1 >> vfat 24576 1 >> fat 90112 1 vfat >> sch_cake 36864 4 >> intel_powerclamp 20480 0 >> psmouse 184320 0 >> pcspkr 16384 0 >> i2c_piix4 28672 0 >> e1000 163840 0 >> i2c_core 106496 2 psmouse,i2c_piix4 >> crct10dif_pclmul 16384 1 >> crc32_pclmul 16384 0 >> ata_generic 16384 0 >> pata_acpi 16384 0 >> ghash_clmulni_intel 16384 0 >> serio_raw 20480 0 >> ohci_pci 20480 0 >> video 57344 0 > > Yes, this looks more like it. > >> The last 8 entries are the same but a whole load of others are missing. > > Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t. > > I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else. > > -Michael > >> >> Regards, >> Adolf. >>>> Tried modprobe e1000 but this came back with the following error >>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service >>>> >>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once. >>>> >>>> Regards >>>> Adolf >>>>>> On 11 May 2022, at 20:08, Adolf Belka adolf.belka@ipfire.org wrote: >>>>>> >>>>>> Hi All, >>>>>> >>>>>> On 11/05/2022 20:48, Jon Murphy wrote: >>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info: >>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* >>>>>>> Shouldn’t this be Core Update 168? >>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases. >>>>>> >>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm. >>>>>> >>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box. >>>>>> >>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box. >>>>>> >>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box. >>>>> That seems to be a bug that should not be there. Did we recently update apache? >>>>> -Michael >>>>>> >>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem. >>>>>> >>>>>> I think I am going to raise a bug on this. >>>>>> >>>>>> Regards, >>>>>> Adolf. >>>>>>> Jon >>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote: >>>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote: >>>>>>>>> Hi Leo, >>>>>>>>> >>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote: >>>>>>>>>> Hi Adolf, >>>>>>>>>> >>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code: >>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that. >>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 >>>>>>>>> Now I see it in the code. Thanks very much. >>>>>>>>>> >>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems! >>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions. >>>>>>>>> >>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. >>>>>>>>> Will wait to hear other inputs on this problem. >>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again. >>>>>>>>> >>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones. >>>>>>>> >>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm. >>>>>>>> >>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/ >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Adolf. >>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Adolf. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Best regards >>>>>>>>>> Leo >>>>>>>>>> >>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>>>>>>>>> Hi All, >>>>>>>>>>> >>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts. >>>>>>>>>>> >>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> First try:- >>>>>>>>>>> >>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had >>>>>>>>>>> >>>>>>>>>>> Core Update 167 Development Build: master/c22d834c >>>>>>>>>>> >>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases. >>>>>>>>>>> >>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped. >>>>>>>>>>> >>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>>>> >>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access. >>>>>>>>>>> >>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message >>>>>>>>>>> >>>>>>>>>>> starting >>>>>>>>>>> DUID 00:04:70:........ >>>>>>>>>>> red0: interface not found >>>>>>>>>>> >>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0. >>>>>>>>>>> >>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same >>>>>>>>>>> >>>>>>>>>>> Loading of module with unavailable key is rejected >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Second try:- >>>>>>>>>>> >>>>>>>>>>> This time running the update took over 10 mins. >>>>>>>>>>> >>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167. >>>>>>>>>>> >>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>>>>>>>>> >>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time. >>>>>>>>>>> >>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28 >>>>>>>>>>> >>>>>>>>>>> Repository was set back to Stable. >>>>>>>>>>> >>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168. >>>>>>>>>>> >>>>>>>>>>> Before running the pakfire directory had the following >>>>>>>>>>> >>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>>>> >>>>>>>>>>> After running it had >>>>>>>>>>> >>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >>>>>>>>>>> >>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen. >>>>>>>>>>> >>>>>>>>>>> Rebooted. >>>>>>>>>>> >>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c >>>>>>>>>>> >>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it >>>>>>>>>>> >>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Third try:- >>>>>>>>>>> >>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>>>> >>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly. >>>>>>>>>>> >>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >>>>>>>>>>> >>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell. >>>>>>>>>>> >>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! >>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce >>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 >>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. >>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 >>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes >>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. >>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. >>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... >>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 >>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... >>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. >>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... >>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 >>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts... >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... >>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing. >>>>>>>>>>> >>>>>>>>>>> Rebooted >>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>>>> Several repeats of the following two lines on the console screen:- >>>>>>>>>>> >>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) >>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I have given up at this point. >>>>>>>>>>> >>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know. >>>>>>>>>>> >>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> Adolf. >
Hello,
On 30 May 2022, at 19:57, Peter Müller peter.mueller@ipfire.org wrote:
Hello Michael,
as promised earlier today:
Hello Adolf,
thank you for all the extensive testing and reporting back.
Ah, indeed, the newly introduced Cron job is stil missing due to silly me confusing the fcrontab binary and an actual fcrontab. I think the most straight-forward way to fix this is to just ship the entire crontab file, and force fcrontab to recreate the
- yes - fcrontab based on its content.
@Michael: How does one ship a single file with a Core Update that is not covered by any rootfile?
You add it to the “files” list?
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=config/rootfiles/core/168/...
I am not sure if I get the question.
- ping - :-)
Thanks, and best regards, Peter Müller
Thanks, and best regards, Peter Müller
Hi All,
Virtually everything I have tested in CU168 looks to be working as expected.
The only problem I have found to-date is the Rules update for Suricata in the Intrusion Prevention System.
The old daily or weekly option is gone and the update should now occur twice per day but that entry is not in the fcrontab in CU168 Testing so no updates are occurring for the rulesets unless you manually force them when the update occurs with no problem.
Regards,
Adolf
On 17/05/2022 10:09, Adolf Belka wrote:
Hi Michael,
On 13/05/2022 22:37, Michael Tremer wrote:
Hello,
Okay, this is really bad then if we are breaking RAID installations.
I tried to debug this, but VirtualBox is not my friend today.
I asked some bug reporter to provide me with the logs that I want over here:
After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
Thanks very much for all the help.
Regards, Adolf.
-Michael
On 13 May 2022, at 18:12, Adolf Belka adolf.belka@ipfire.org wrote:
Hi All,
I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
The vm shows the following
-bash-5.1$ cat /proc/mdstat Personalities : md127 : inactive sdb[1](S) 52428724 blocks super 1.0
unused devices: <none>
It looks like IPFire was only running on sda.
So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
The CU168 vm that is working has the same as the above setting.
The CU168 vm that is not working has the following setting
-bash-5.1$ cat /proc/mdstat Personalities : md127 : inactive sda[0](S) 52428724 blocks super 1.0
unused devices: <none>
So it looks like it has booted up on the disk that was not updated from the raid pair.
This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
It is probably dependent on which of the two disks actually get used to boot from.
So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
So raid systems need to have their raid arrays fully working before doing an upgrade.
At least the above is my interpretation of what I have found. What does everyone else think.
Regards,
Adolf.
On 13/05/2022 15:59, Adolf Belka wrote: > Hi Michael, > > I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules. > > I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot. > > However I did copy the boot directory info for these two cases and found an interesting effect. > > For the problem vm here was the boot directory layout of the CU167 base machine > > drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . > drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. > -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire > drwxr-xr-x 3 root root 16K Jan 1 1970 efi > drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub > -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img > drwx------ 2 root root 16K Sep 30 2021 lost+found > -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire > -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire > > After upgrade and before reboot this had changed to > > drwxr-xr-x 5 root root 4.0K May 13 15:40 . > drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. > -rw-r--r-- 1 root root 179K May 3 04:02 config-5.15.35-ipfire > drwxr-xr-x 3 root root 16K Jan 1 1970 efi > drwxr-xr-x 6 root root 4.0K May 13 15:41 grub > -rw------- 1 root root 26M May 13 15:41 initramfs-5.15.35-ipfire.img > drwx------ 2 root root 16K Sep 30 2021 lost+found > -rw-r--r-- 1 root root 4.7M May 3 04:02 System.map-5.15.35-ipfire > -rw-r--r-- 1 root root 6.7M May 3 04:02 vmlinuz-5.15.35-ipfire > > Dates and times of some of the files/directories have changed as would be expected. > > After Reboot and with the problem the boot directory was now > > drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . > drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. > -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire > drwxr-xr-x 3 root root 4.0K Sep 30 2021 efi > drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub > -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img > drwx------ 2 root root 16K Sep 30 2021 lost+found > -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire > -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire > > Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed > > For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade. > > I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not. > > Regards, > > Adolf. > > On 13/05/2022 12:09, Michael Tremer wrote: >> Hello, >> >>> On 12 May 2022, at 21:10, Adolf Belka adolf.belka@ipfire.org wrote: >>> >>> Hi Michael, >>> >>> On 12/05/2022 14:53, Michael Tremer wrote: >>>> Hello, >>>>> On 12 May 2022, at 12:25, Adolf Belka adolf.belka@ipfire.org wrote: >>>>> >>>>> Hi, >>>>> >>>>> On 12/05/2022 11:13, Michael Tremer wrote: >>>>>> Hello, >>>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while. >>>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved. >>>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started? >>>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that. >>>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm. >>>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case? >>>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces. >>>> Okay. That is indeed quite bad then. >>>> Since there is no kernel in 168, is there a chance that we broke the update to 167? >>> I don't know but as far as I have been concerned CU167 has worked well on my vm. >>> Is there something I should look for on my CU167 vm? >>>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface. >>>> Third time lucky isn’t good enough for me :) >>> Me neither :-) >>>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces. >>>>> >>>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers >>>> Have there been any modules loaded? If one loads, they all should load. >>> This is what I get with lsmod >>> >>> Module Size Used by >>> crct10dif_pclmul 16384 1 >>> crc32_pclmul 16384 0 >>> ghash_clmulni_intel 16384 0 >>> serio_raw 20480 0 >>> ohci_pci 20480 0 >>> ata_generic 16384 0 >>> pata_acpi 16384 0 >>> video 57344 0 >> >> It looks like these modules could have come from the initramdisk image. At least that is working well. >> >>> On my running CU1678 vm lsmod gives the following >>> >>> Module Size Used by >>> tun 61440 2 >>> nfnetlink_queue 28672 1 >>> xt_NFQUEUE 16384 8 >>> xt_MASQUERADE 20480 1 >>> cfg80211 1036288 0 >>> rfkill 32768 1 cfg80211 >>> 8021q 40960 0 >>> garp 16384 1 8021q >>> xt_set 16384 260 >>> ip_set_hash_net 49152 258 >>> ip_set 57344 2 xt_set,ip_set_hash_net >>> xt_hashlimit 20480 2 >>> xt_multiport 20480 4 >>> xt_policy 16384 5 >>> xt_TCPMSS 16384 1 >>> xt_conntrack 16384 7 >>> xt_comment 16384 18 >>> ipt_REJECT 16384 1 >>> nf_reject_ipv4 16384 1 ipt_REJECT >>> xt_LOG 20480 26 >>> xt_limit 16384 25 >>> xt_mark 16384 27 >>> xt_connmark 16384 2 >>> nf_log_syslog 24576 26 >>> iptable_raw 16384 0 >>> iptable_mangle 16384 1 >>> iptable_filter 16384 1 >>> vfat 24576 1 >>> fat 90112 1 vfat >>> sch_cake 36864 4 >>> intel_powerclamp 20480 0 >>> psmouse 184320 0 >>> pcspkr 16384 0 >>> i2c_piix4 28672 0 >>> e1000 163840 0 >>> i2c_core 106496 2 psmouse,i2c_piix4 >>> crct10dif_pclmul 16384 1 >>> crc32_pclmul 16384 0 >>> ata_generic 16384 0 >>> pata_acpi 16384 0 >>> ghash_clmulni_intel 16384 0 >>> serio_raw 20480 0 >>> ohci_pci 20480 0 >>> video 57344 0 >> >> Yes, this looks more like it. >> >>> The last 8 entries are the same but a whole load of others are missing. >> >> Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t. >> >> I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else. >> >> -Michael >> >>> >>> Regards, >>> Adolf. >>>>> Tried modprobe e1000 but this came back with the following error >>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service >>>>> >>>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once. >>>>> >>>>> Regards >>>>> Adolf >>>>>>> On 11 May 2022, at 20:08, Adolf Belka adolf.belka@ipfire.org wrote: >>>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> On 11/05/2022 20:48, Jon Murphy wrote: >>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info: >>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* >>>>>>>> Shouldn’t this be Core Update 168? >>>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases. >>>>>>> >>>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm. >>>>>>> >>>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box. >>>>>>> >>>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box. >>>>>>> >>>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box. >>>>>> That seems to be a bug that should not be there. Did we recently update apache? >>>>>> -Michael >>>>>>> >>>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem. >>>>>>> >>>>>>> I think I am going to raise a bug on this. >>>>>>> >>>>>>> Regards, >>>>>>> Adolf. >>>>>>>> Jon >>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote: >>>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote: >>>>>>>>>> Hi Leo, >>>>>>>>>> >>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote: >>>>>>>>>>> Hi Adolf, >>>>>>>>>>> >>>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code: >>>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that. >>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 >>>>>>>>>> Now I see it in the code. Thanks very much. >>>>>>>>>>> >>>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems! >>>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions. >>>>>>>>>> >>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. >>>>>>>>>> Will wait to hear other inputs on this problem. >>>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again. >>>>>>>>>> >>>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones. >>>>>>>>> >>>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm. >>>>>>>>> >>>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/ >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Adolf. >>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Adolf. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Best regards >>>>>>>>>>> Leo >>>>>>>>>>> >>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>>>>>>>>>> Hi All, >>>>>>>>>>>> >>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts. >>>>>>>>>>>> >>>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> First try:- >>>>>>>>>>>> >>>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had >>>>>>>>>>>> >>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c >>>>>>>>>>>> >>>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases. >>>>>>>>>>>> >>>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped. >>>>>>>>>>>> >>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>>>>> >>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access. >>>>>>>>>>>> >>>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message >>>>>>>>>>>> >>>>>>>>>>>> starting >>>>>>>>>>>> DUID 00:04:70:........ >>>>>>>>>>>> red0: interface not found >>>>>>>>>>>> >>>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0. >>>>>>>>>>>> >>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same >>>>>>>>>>>> >>>>>>>>>>>> Loading of module with unavailable key is rejected >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Second try:- >>>>>>>>>>>> >>>>>>>>>>>> This time running the update took over 10 mins. >>>>>>>>>>>> >>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167. >>>>>>>>>>>> >>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>>>>>>>>>> >>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time. >>>>>>>>>>>> >>>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28 >>>>>>>>>>>> >>>>>>>>>>>> Repository was set back to Stable. >>>>>>>>>>>> >>>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168. >>>>>>>>>>>> >>>>>>>>>>>> Before running the pakfire directory had the following >>>>>>>>>>>> >>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>>>>> >>>>>>>>>>>> After running it had >>>>>>>>>>>> >>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >>>>>>>>>>>> >>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen. >>>>>>>>>>>> >>>>>>>>>>>> Rebooted. >>>>>>>>>>>> >>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c >>>>>>>>>>>> >>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it >>>>>>>>>>>> >>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Third try:- >>>>>>>>>>>> >>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>>>>> >>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly. >>>>>>>>>>>> >>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >>>>>>>>>>>> >>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell. >>>>>>>>>>>> >>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! >>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce >>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 >>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. >>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 >>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes >>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. >>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. >>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... >>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 >>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. >>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... >>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 >>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing. >>>>>>>>>>>> >>>>>>>>>>>> Rebooted >>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>>>>> Several repeats of the following two lines on the console screen:- >>>>>>>>>>>> >>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) >>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I have given up at this point. >>>>>>>>>>>> >>>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know. >>>>>>>>>>>> >>>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> >>>>>>>>>>>> Adolf. >>
Hello Michael,
Hello,
On 30 May 2022, at 19:57, Peter Müller peter.mueller@ipfire.org wrote:
Hello Michael,
as promised earlier today:
Hello Adolf,
thank you for all the extensive testing and reporting back.
Ah, indeed, the newly introduced Cron job is stil missing due to silly me confusing the fcrontab binary and an actual fcrontab. I think the most straight-forward way to fix this is to just ship the entire crontab file, and force fcrontab to recreate the
- yes - fcrontab based on its content.
@Michael: How does one ship a single file with a Core Update that is not covered by any rootfile?
You add it to the “files” list?
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=config/rootfiles/core/168/...
I am not sure if I get the question.
my problem is as follows: With the IDSv4 changes, a new cron job has been introduced, and an old one needs to be purged from the crontab. Smilar to what we do in https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/fcron;h=484deb63d77872..., I would therefore like to ship the entire config/cron/crontab file, and load it into fcron the same way.
Since config/cron/crontab is not part of any rootfile, I am seeking a way to ship it with the Core Update. :-)
Thanks, and best regards, Peter Müller
- ping - :-)
Thanks, and best regards, Peter Müller
Thanks, and best regards, Peter Müller
Hi All,
Virtually everything I have tested in CU168 looks to be working as expected.
The only problem I have found to-date is the Rules update for Suricata in the Intrusion Prevention System.
The old daily or weekly option is gone and the update should now occur twice per day but that entry is not in the fcrontab in CU168 Testing so no updates are occurring for the rulesets unless you manually force them when the update occurs with no problem.
Regards,
Adolf
On 17/05/2022 10:09, Adolf Belka wrote:
Hi Michael,
On 13/05/2022 22:37, Michael Tremer wrote:
Hello,
Okay, this is really bad then if we are breaking RAID installations.
I tried to debug this, but VirtualBox is not my friend today.
I asked some bug reporter to provide me with the logs that I want over here:
After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
Thanks very much for all the help.
Regards, Adolf.
-Michael
> On 13 May 2022, at 18:12, Adolf Belka adolf.belka@ipfire.org wrote: > > Hi All, > > I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go. > > I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk. > > I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array. > > The vm shows the following > > -bash-5.1$ cat /proc/mdstat > Personalities : > md127 : inactive sdb[1](S) > 52428724 blocks super 1.0 > > unused devices: <none> > > > It looks like IPFire was only running on sda. > > So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated. > > The CU168 vm that is working has the same as the above setting. > > The CU168 vm that is not working has the following setting > > -bash-5.1$ cat /proc/mdstat > Personalities : > md127 : inactive sda[0](S) > 52428724 blocks super 1.0 > > unused devices: <none> > > So it looks like it has booted up on the disk that was not updated from the raid pair. > > > This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode. > > It is probably dependent on which of the two disks actually get used to boot from. > > So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade. > > So raid systems need to have their raid arrays fully working before doing an upgrade. > > At least the above is my interpretation of what I have found. What does everyone else think. > > > Regards, > > Adolf. > > > On 13/05/2022 15:59, Adolf Belka wrote: >> Hi Michael, >> >> I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules. >> >> I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot. >> >> However I did copy the boot directory info for these two cases and found an interesting effect. >> >> For the problem vm here was the boot directory layout of the CU167 base machine >> >> drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . >> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. >> -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire >> drwxr-xr-x 3 root root 16K Jan 1 1970 efi >> drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub >> -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img >> drwx------ 2 root root 16K Sep 30 2021 lost+found >> -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire >> -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire >> >> After upgrade and before reboot this had changed to >> >> drwxr-xr-x 5 root root 4.0K May 13 15:40 . >> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. >> -rw-r--r-- 1 root root 179K May 3 04:02 config-5.15.35-ipfire >> drwxr-xr-x 3 root root 16K Jan 1 1970 efi >> drwxr-xr-x 6 root root 4.0K May 13 15:41 grub >> -rw------- 1 root root 26M May 13 15:41 initramfs-5.15.35-ipfire.img >> drwx------ 2 root root 16K Sep 30 2021 lost+found >> -rw-r--r-- 1 root root 4.7M May 3 04:02 System.map-5.15.35-ipfire >> -rw-r--r-- 1 root root 6.7M May 3 04:02 vmlinuz-5.15.35-ipfire >> >> Dates and times of some of the files/directories have changed as would be expected. >> >> After Reboot and with the problem the boot directory was now >> >> drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . >> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. >> -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire >> drwxr-xr-x 3 root root 4.0K Sep 30 2021 efi >> drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub >> -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img >> drwx------ 2 root root 16K Sep 30 2021 lost+found >> -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire >> -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire >> >> Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed >> >> For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade. >> >> I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not. >> >> Regards, >> >> Adolf. >> >> On 13/05/2022 12:09, Michael Tremer wrote: >>> Hello, >>> >>>> On 12 May 2022, at 21:10, Adolf Belka adolf.belka@ipfire.org wrote: >>>> >>>> Hi Michael, >>>> >>>> On 12/05/2022 14:53, Michael Tremer wrote: >>>>> Hello, >>>>>> On 12 May 2022, at 12:25, Adolf Belka adolf.belka@ipfire.org wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> On 12/05/2022 11:13, Michael Tremer wrote: >>>>>>> Hello, >>>>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while. >>>>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved. >>>>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started? >>>>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that. >>>>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm. >>>>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case? >>>>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces. >>>>> Okay. That is indeed quite bad then. >>>>> Since there is no kernel in 168, is there a chance that we broke the update to 167? >>>> I don't know but as far as I have been concerned CU167 has worked well on my vm. >>>> Is there something I should look for on my CU167 vm? >>>>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface. >>>>> Third time lucky isn’t good enough for me :) >>>> Me neither :-) >>>>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces. >>>>>> >>>>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers >>>>> Have there been any modules loaded? If one loads, they all should load. >>>> This is what I get with lsmod >>>> >>>> Module Size Used by >>>> crct10dif_pclmul 16384 1 >>>> crc32_pclmul 16384 0 >>>> ghash_clmulni_intel 16384 0 >>>> serio_raw 20480 0 >>>> ohci_pci 20480 0 >>>> ata_generic 16384 0 >>>> pata_acpi 16384 0 >>>> video 57344 0 >>> >>> It looks like these modules could have come from the initramdisk image. At least that is working well. >>> >>>> On my running CU1678 vm lsmod gives the following >>>> >>>> Module Size Used by >>>> tun 61440 2 >>>> nfnetlink_queue 28672 1 >>>> xt_NFQUEUE 16384 8 >>>> xt_MASQUERADE 20480 1 >>>> cfg80211 1036288 0 >>>> rfkill 32768 1 cfg80211 >>>> 8021q 40960 0 >>>> garp 16384 1 8021q >>>> xt_set 16384 260 >>>> ip_set_hash_net 49152 258 >>>> ip_set 57344 2 xt_set,ip_set_hash_net >>>> xt_hashlimit 20480 2 >>>> xt_multiport 20480 4 >>>> xt_policy 16384 5 >>>> xt_TCPMSS 16384 1 >>>> xt_conntrack 16384 7 >>>> xt_comment 16384 18 >>>> ipt_REJECT 16384 1 >>>> nf_reject_ipv4 16384 1 ipt_REJECT >>>> xt_LOG 20480 26 >>>> xt_limit 16384 25 >>>> xt_mark 16384 27 >>>> xt_connmark 16384 2 >>>> nf_log_syslog 24576 26 >>>> iptable_raw 16384 0 >>>> iptable_mangle 16384 1 >>>> iptable_filter 16384 1 >>>> vfat 24576 1 >>>> fat 90112 1 vfat >>>> sch_cake 36864 4 >>>> intel_powerclamp 20480 0 >>>> psmouse 184320 0 >>>> pcspkr 16384 0 >>>> i2c_piix4 28672 0 >>>> e1000 163840 0 >>>> i2c_core 106496 2 psmouse,i2c_piix4 >>>> crct10dif_pclmul 16384 1 >>>> crc32_pclmul 16384 0 >>>> ata_generic 16384 0 >>>> pata_acpi 16384 0 >>>> ghash_clmulni_intel 16384 0 >>>> serio_raw 20480 0 >>>> ohci_pci 20480 0 >>>> video 57344 0 >>> >>> Yes, this looks more like it. >>> >>>> The last 8 entries are the same but a whole load of others are missing. >>> >>> Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t. >>> >>> I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else. >>> >>> -Michael >>> >>>> >>>> Regards, >>>> Adolf. >>>>>> Tried modprobe e1000 but this came back with the following error >>>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service >>>>>> >>>>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once. >>>>>> >>>>>> Regards >>>>>> Adolf >>>>>>>> On 11 May 2022, at 20:08, Adolf Belka adolf.belka@ipfire.org wrote: >>>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> On 11/05/2022 20:48, Jon Murphy wrote: >>>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info: >>>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* >>>>>>>>> Shouldn’t this be Core Update 168? >>>>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases. >>>>>>>> >>>>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm. >>>>>>>> >>>>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box. >>>>>>>> >>>>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box. >>>>>>>> >>>>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box. >>>>>>> That seems to be a bug that should not be there. Did we recently update apache? >>>>>>> -Michael >>>>>>>> >>>>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem. >>>>>>>> >>>>>>>> I think I am going to raise a bug on this. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Adolf. >>>>>>>>> Jon >>>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote: >>>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote: >>>>>>>>>>> Hi Leo, >>>>>>>>>>> >>>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote: >>>>>>>>>>>> Hi Adolf, >>>>>>>>>>>> >>>>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code: >>>>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that. >>>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 >>>>>>>>>>> Now I see it in the code. Thanks very much. >>>>>>>>>>>> >>>>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems! >>>>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions. >>>>>>>>>>> >>>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. >>>>>>>>>>> Will wait to hear other inputs on this problem. >>>>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again. >>>>>>>>>>> >>>>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones. >>>>>>>>>> >>>>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm. >>>>>>>>>> >>>>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/ >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Adolf. >>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Adolf. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Best regards >>>>>>>>>>>> Leo >>>>>>>>>>>> >>>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>>>>>>>>>>> Hi All, >>>>>>>>>>>>> >>>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts. >>>>>>>>>>>>> >>>>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> First try:- >>>>>>>>>>>>> >>>>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had >>>>>>>>>>>>> >>>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c >>>>>>>>>>>>> >>>>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases. >>>>>>>>>>>>> >>>>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped. >>>>>>>>>>>>> >>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>>>>>> >>>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access. >>>>>>>>>>>>> >>>>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message >>>>>>>>>>>>> >>>>>>>>>>>>> starting >>>>>>>>>>>>> DUID 00:04:70:........ >>>>>>>>>>>>> red0: interface not found >>>>>>>>>>>>> >>>>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0. >>>>>>>>>>>>> >>>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same >>>>>>>>>>>>> >>>>>>>>>>>>> Loading of module with unavailable key is rejected >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Second try:- >>>>>>>>>>>>> >>>>>>>>>>>>> This time running the update took over 10 mins. >>>>>>>>>>>>> >>>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167. >>>>>>>>>>>>> >>>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>>>>>>>>>>> >>>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time. >>>>>>>>>>>>> >>>>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28 >>>>>>>>>>>>> >>>>>>>>>>>>> Repository was set back to Stable. >>>>>>>>>>>>> >>>>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168. >>>>>>>>>>>>> >>>>>>>>>>>>> Before running the pakfire directory had the following >>>>>>>>>>>>> >>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>>>>>> >>>>>>>>>>>>> After running it had >>>>>>>>>>>>> >>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >>>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >>>>>>>>>>>>> >>>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen. >>>>>>>>>>>>> >>>>>>>>>>>>> Rebooted. >>>>>>>>>>>>> >>>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c >>>>>>>>>>>>> >>>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it >>>>>>>>>>>>> >>>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Third try:- >>>>>>>>>>>>> >>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>>>>>> >>>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly. >>>>>>>>>>>>> >>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >>>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >>>>>>>>>>>>> >>>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell. >>>>>>>>>>>>> >>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! >>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce >>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 >>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. >>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 >>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes >>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. >>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. >>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... >>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 >>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. >>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... >>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 >>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing. >>>>>>>>>>>>> >>>>>>>>>>>>> Rebooted >>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>>>>>> Several repeats of the following two lines on the console screen:- >>>>>>>>>>>>> >>>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) >>>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I have given up at this point. >>>>>>>>>>>>> >>>>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know. >>>>>>>>>>>>> >>>>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> >>>>>>>>>>>>> Adolf. >>>
Hello,
On 31 May 2022, at 18:36, Peter Müller peter.mueller@ipfire.org wrote:
Hello Michael,
Hello,
On 30 May 2022, at 19:57, Peter Müller peter.mueller@ipfire.org wrote:
Hello Michael,
as promised earlier today:
Hello Adolf,
thank you for all the extensive testing and reporting back.
Ah, indeed, the newly introduced Cron job is stil missing due to silly me confusing the fcrontab binary and an actual fcrontab. I think the most straight-forward way to fix this is to just ship the entire crontab file, and force fcrontab to recreate the
- yes - fcrontab based on its content.
@Michael: How does one ship a single file with a Core Update that is not covered by any rootfile?
You add it to the “files” list?
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=config/rootfiles/core/168/...
I am not sure if I get the question.
my problem is as follows: With the IDSv4 changes, a new cron job has been introduced, and an old one needs to be purged from the crontab. Smilar to what we do in https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/fcron;h=484deb63d77872..., I would therefore like to ship the entire config/cron/crontab file, and load it into fcron the same way.
Since config/cron/crontab is not part of any rootfile, I am seeking a way to ship it with the Core Update. :-)
It is: https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=config/rootfiles/common/fc...
It is in /var/spool/cron/root.orig.
That is the file that you will need to ship and then you will have to call “fcrontab -z” so that it will be converted into the binary format that fcron uses.
Just note that any custom cron jobs that users might have created in there will be lost. That is why we normally only edit that file.
Thanks, and best regards, Peter Müller
- ping - :-)
Thanks, and best regards, Peter Müller
Thanks, and best regards, Peter Müller
Hi All,
Virtually everything I have tested in CU168 looks to be working as expected.
The only problem I have found to-date is the Rules update for Suricata in the Intrusion Prevention System.
The old daily or weekly option is gone and the update should now occur twice per day but that entry is not in the fcrontab in CU168 Testing so no updates are occurring for the rulesets unless you manually force them when the update occurs with no problem.
Regards,
Adolf
On 17/05/2022 10:09, Adolf Belka wrote:
Hi Michael,
On 13/05/2022 22:37, Michael Tremer wrote: > Hello, > > Okay, this is really bad then if we are breaking RAID installations. > > I tried to debug this, but VirtualBox is not my friend today. > > I asked some bug reporter to provide me with the logs that I want over here: > > https://bugzilla.ipfire.org/show_bug.cgi?id=12862 After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
Thanks very much for all the help.
Regards, Adolf. > > -Michael > >> On 13 May 2022, at 18:12, Adolf Belka adolf.belka@ipfire.org wrote: >> >> Hi All, >> >> I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go. >> >> I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk. >> >> I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array. >> >> The vm shows the following >> >> -bash-5.1$ cat /proc/mdstat >> Personalities : >> md127 : inactive sdb[1](S) >> 52428724 blocks super 1.0 >> >> unused devices: <none> >> >> >> It looks like IPFire was only running on sda. >> >> So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated. >> >> The CU168 vm that is working has the same as the above setting. >> >> The CU168 vm that is not working has the following setting >> >> -bash-5.1$ cat /proc/mdstat >> Personalities : >> md127 : inactive sda[0](S) >> 52428724 blocks super 1.0 >> >> unused devices: <none> >> >> So it looks like it has booted up on the disk that was not updated from the raid pair. >> >> >> This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode. >> >> It is probably dependent on which of the two disks actually get used to boot from. >> >> So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade. >> >> So raid systems need to have their raid arrays fully working before doing an upgrade. >> >> At least the above is my interpretation of what I have found. What does everyone else think. >> >> >> Regards, >> >> Adolf. >> >> >> On 13/05/2022 15:59, Adolf Belka wrote: >>> Hi Michael, >>> >>> I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules. >>> >>> I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot. >>> >>> However I did copy the boot directory info for these two cases and found an interesting effect. >>> >>> For the problem vm here was the boot directory layout of the CU167 base machine >>> >>> drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . >>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. >>> -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire >>> drwxr-xr-x 3 root root 16K Jan 1 1970 efi >>> drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub >>> -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img >>> drwx------ 2 root root 16K Sep 30 2021 lost+found >>> -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire >>> -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire >>> >>> After upgrade and before reboot this had changed to >>> >>> drwxr-xr-x 5 root root 4.0K May 13 15:40 . >>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. >>> -rw-r--r-- 1 root root 179K May 3 04:02 config-5.15.35-ipfire >>> drwxr-xr-x 3 root root 16K Jan 1 1970 efi >>> drwxr-xr-x 6 root root 4.0K May 13 15:41 grub >>> -rw------- 1 root root 26M May 13 15:41 initramfs-5.15.35-ipfire.img >>> drwx------ 2 root root 16K Sep 30 2021 lost+found >>> -rw-r--r-- 1 root root 4.7M May 3 04:02 System.map-5.15.35-ipfire >>> -rw-r--r-- 1 root root 6.7M May 3 04:02 vmlinuz-5.15.35-ipfire >>> >>> Dates and times of some of the files/directories have changed as would be expected. >>> >>> After Reboot and with the problem the boot directory was now >>> >>> drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . >>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. >>> -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire >>> drwxr-xr-x 3 root root 4.0K Sep 30 2021 efi >>> drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub >>> -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img >>> drwx------ 2 root root 16K Sep 30 2021 lost+found >>> -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire >>> -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire >>> >>> Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed >>> >>> For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade. >>> >>> I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not. >>> >>> Regards, >>> >>> Adolf. >>> >>> On 13/05/2022 12:09, Michael Tremer wrote: >>>> Hello, >>>> >>>>> On 12 May 2022, at 21:10, Adolf Belka adolf.belka@ipfire.org wrote: >>>>> >>>>> Hi Michael, >>>>> >>>>> On 12/05/2022 14:53, Michael Tremer wrote: >>>>>> Hello, >>>>>>> On 12 May 2022, at 12:25, Adolf Belka adolf.belka@ipfire.org wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> On 12/05/2022 11:13, Michael Tremer wrote: >>>>>>>> Hello, >>>>>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while. >>>>>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved. >>>>>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started? >>>>>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that. >>>>>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm. >>>>>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case? >>>>>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces. >>>>>> Okay. That is indeed quite bad then. >>>>>> Since there is no kernel in 168, is there a chance that we broke the update to 167? >>>>> I don't know but as far as I have been concerned CU167 has worked well on my vm. >>>>> Is there something I should look for on my CU167 vm? >>>>>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface. >>>>>> Third time lucky isn’t good enough for me :) >>>>> Me neither :-) >>>>>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces. >>>>>>> >>>>>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers >>>>>> Have there been any modules loaded? If one loads, they all should load. >>>>> This is what I get with lsmod >>>>> >>>>> Module Size Used by >>>>> crct10dif_pclmul 16384 1 >>>>> crc32_pclmul 16384 0 >>>>> ghash_clmulni_intel 16384 0 >>>>> serio_raw 20480 0 >>>>> ohci_pci 20480 0 >>>>> ata_generic 16384 0 >>>>> pata_acpi 16384 0 >>>>> video 57344 0 >>>> >>>> It looks like these modules could have come from the initramdisk image. At least that is working well. >>>> >>>>> On my running CU1678 vm lsmod gives the following >>>>> >>>>> Module Size Used by >>>>> tun 61440 2 >>>>> nfnetlink_queue 28672 1 >>>>> xt_NFQUEUE 16384 8 >>>>> xt_MASQUERADE 20480 1 >>>>> cfg80211 1036288 0 >>>>> rfkill 32768 1 cfg80211 >>>>> 8021q 40960 0 >>>>> garp 16384 1 8021q >>>>> xt_set 16384 260 >>>>> ip_set_hash_net 49152 258 >>>>> ip_set 57344 2 xt_set,ip_set_hash_net >>>>> xt_hashlimit 20480 2 >>>>> xt_multiport 20480 4 >>>>> xt_policy 16384 5 >>>>> xt_TCPMSS 16384 1 >>>>> xt_conntrack 16384 7 >>>>> xt_comment 16384 18 >>>>> ipt_REJECT 16384 1 >>>>> nf_reject_ipv4 16384 1 ipt_REJECT >>>>> xt_LOG 20480 26 >>>>> xt_limit 16384 25 >>>>> xt_mark 16384 27 >>>>> xt_connmark 16384 2 >>>>> nf_log_syslog 24576 26 >>>>> iptable_raw 16384 0 >>>>> iptable_mangle 16384 1 >>>>> iptable_filter 16384 1 >>>>> vfat 24576 1 >>>>> fat 90112 1 vfat >>>>> sch_cake 36864 4 >>>>> intel_powerclamp 20480 0 >>>>> psmouse 184320 0 >>>>> pcspkr 16384 0 >>>>> i2c_piix4 28672 0 >>>>> e1000 163840 0 >>>>> i2c_core 106496 2 psmouse,i2c_piix4 >>>>> crct10dif_pclmul 16384 1 >>>>> crc32_pclmul 16384 0 >>>>> ata_generic 16384 0 >>>>> pata_acpi 16384 0 >>>>> ghash_clmulni_intel 16384 0 >>>>> serio_raw 20480 0 >>>>> ohci_pci 20480 0 >>>>> video 57344 0 >>>> >>>> Yes, this looks more like it. >>>> >>>>> The last 8 entries are the same but a whole load of others are missing. >>>> >>>> Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t. >>>> >>>> I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else. >>>> >>>> -Michael >>>> >>>>> >>>>> Regards, >>>>> Adolf. >>>>>>> Tried modprobe e1000 but this came back with the following error >>>>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service >>>>>>> >>>>>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once. >>>>>>> >>>>>>> Regards >>>>>>> Adolf >>>>>>>>> On 11 May 2022, at 20:08, Adolf Belka adolf.belka@ipfire.org wrote: >>>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> On 11/05/2022 20:48, Jon Murphy wrote: >>>>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info: >>>>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* >>>>>>>>>> Shouldn’t this be Core Update 168? >>>>>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases. >>>>>>>>> >>>>>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm. >>>>>>>>> >>>>>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box. >>>>>>>>> >>>>>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box. >>>>>>>>> >>>>>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box. >>>>>>>> That seems to be a bug that should not be there. Did we recently update apache? >>>>>>>> -Michael >>>>>>>>> >>>>>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem. >>>>>>>>> >>>>>>>>> I think I am going to raise a bug on this. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Adolf. >>>>>>>>>> Jon >>>>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi All, >>>>>>>>>>> >>>>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote: >>>>>>>>>>>> Hi Leo, >>>>>>>>>>>> >>>>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote: >>>>>>>>>>>>> Hi Adolf, >>>>>>>>>>>>> >>>>>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code: >>>>>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that. >>>>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 >>>>>>>>>>>> Now I see it in the code. Thanks very much. >>>>>>>>>>>>> >>>>>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems! >>>>>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions. >>>>>>>>>>>> >>>>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. >>>>>>>>>>>> Will wait to hear other inputs on this problem. >>>>>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again. >>>>>>>>>>>> >>>>>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones. >>>>>>>>>>> >>>>>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm. >>>>>>>>>>> >>>>>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/ >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> Adolf. >>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Adolf. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Best regards >>>>>>>>>>>>> Leo >>>>>>>>>>>>> >>>>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> First try:- >>>>>>>>>>>>>> >>>>>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had >>>>>>>>>>>>>> >>>>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c >>>>>>>>>>>>>> >>>>>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message >>>>>>>>>>>>>> >>>>>>>>>>>>>> starting >>>>>>>>>>>>>> DUID 00:04:70:........ >>>>>>>>>>>>>> red0: interface not found >>>>>>>>>>>>>> >>>>>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same >>>>>>>>>>>>>> >>>>>>>>>>>>>> Loading of module with unavailable key is rejected >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Second try:- >>>>>>>>>>>>>> >>>>>>>>>>>>>> This time running the update took over 10 mins. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167. >>>>>>>>>>>>>> >>>>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Repository was set back to Stable. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Before running the pakfire directory had the following >>>>>>>>>>>>>> >>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>>>>>>> >>>>>>>>>>>>>> After running it had >>>>>>>>>>>>>> >>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >>>>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >>>>>>>>>>>>>> >>>>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Rebooted. >>>>>>>>>>>>>> >>>>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c >>>>>>>>>>>>>> >>>>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it >>>>>>>>>>>>>> >>>>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Third try:- >>>>>>>>>>>>>> >>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>>>>>>> >>>>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly. >>>>>>>>>>>>>> >>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >>>>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >>>>>>>>>>>>>> >>>>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell. >>>>>>>>>>>>>> >>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! >>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce >>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 >>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. >>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 >>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes >>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. >>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. >>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... >>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 >>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. >>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... >>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 >>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Rebooted >>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>>>>>>> Several repeats of the following two lines on the console screen:- >>>>>>>>>>>>>> >>>>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) >>>>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I have given up at this point. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know. >>>>>>>>>>>>>> >>>>>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Adolf.
Hello Michael,
thanks for your reply.
Hello,
On 31 May 2022, at 18:36, Peter Müller peter.mueller@ipfire.org wrote:
Hello Michael,
Hello,
On 30 May 2022, at 19:57, Peter Müller peter.mueller@ipfire.org wrote:
Hello Michael,
as promised earlier today:
Hello Adolf,
thank you for all the extensive testing and reporting back.
Ah, indeed, the newly introduced Cron job is stil missing due to silly me confusing the fcrontab binary and an actual fcrontab. I think the most straight-forward way to fix this is to just ship the entire crontab file, and force fcrontab to recreate the
- yes - fcrontab based on its content.
@Michael: How does one ship a single file with a Core Update that is not covered by any rootfile?
You add it to the “files” list?
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=config/rootfiles/core/168/...
I am not sure if I get the question.
my problem is as follows: With the IDSv4 changes, a new cron job has been introduced, and an old one needs to be purged from the crontab. Smilar to what we do in https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/fcron;h=484deb63d77872..., I would therefore like to ship the entire config/cron/crontab file, and load it into fcron the same way.
Since config/cron/crontab is not part of any rootfile, I am seeking a way to ship it with the Core Update. :-)
It is: https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=config/rootfiles/common/fc...
It is in /var/spool/cron/root.orig.
That is the file that you will need to ship and then you will have to call “fcrontab -z” so that it will be converted into the binary format that fcron uses.
Ah, I see, thanks for the explanation.
https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=4a4fc8f19a8734a7d92895da... now ships this file and rebuilds fcrontab from scratch while upgrading.
Just note that any custom cron jobs that users might have created in there will be lost. That is why we normally only edit that file.
I am willing to take that risk. We need IPFire users using RAID to have a manual change at their systems anyway, and rather than doing some error-prone sed'ing over fcrontab, I think it is better to reset it to a defined state and add a note regarding this to the changelog for Core Update 168.
The latter is now complete from my point of view. As discussed, I will prepare the release announcement and pass it on to you for the final steps later today.
Thanks, and best regards, Peter Müller
Thanks, and best regards, Peter Müller
- ping - :-)
Thanks, and best regards, Peter Müller
Thanks, and best regards, Peter Müller
Hi All,
Virtually everything I have tested in CU168 looks to be working as expected.
The only problem I have found to-date is the Rules update for Suricata in the Intrusion Prevention System.
The old daily or weekly option is gone and the update should now occur twice per day but that entry is not in the fcrontab in CU168 Testing so no updates are occurring for the rulesets unless you manually force them when the update occurs with no problem.
Regards,
Adolf
On 17/05/2022 10:09, Adolf Belka wrote: > Hi Michael, > > On 13/05/2022 22:37, Michael Tremer wrote: >> Hello, >> >> Okay, this is really bad then if we are breaking RAID installations. >> >> I tried to debug this, but VirtualBox is not my friend today. >> >> I asked some bug reporter to provide me with the logs that I want over here: >> >> https://bugzilla.ipfire.org/show_bug.cgi?id=12862 > After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone. > > My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation. > > Thanks very much for all the help. > > Regards, > Adolf. >> >> -Michael >> >>> On 13 May 2022, at 18:12, Adolf Belka adolf.belka@ipfire.org wrote: >>> >>> Hi All, >>> >>> I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go. >>> >>> I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk. >>> >>> I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array. >>> >>> The vm shows the following >>> >>> -bash-5.1$ cat /proc/mdstat >>> Personalities : >>> md127 : inactive sdb[1](S) >>> 52428724 blocks super 1.0 >>> >>> unused devices: <none> >>> >>> >>> It looks like IPFire was only running on sda. >>> >>> So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated. >>> >>> The CU168 vm that is working has the same as the above setting. >>> >>> The CU168 vm that is not working has the following setting >>> >>> -bash-5.1$ cat /proc/mdstat >>> Personalities : >>> md127 : inactive sda[0](S) >>> 52428724 blocks super 1.0 >>> >>> unused devices: <none> >>> >>> So it looks like it has booted up on the disk that was not updated from the raid pair. >>> >>> >>> This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode. >>> >>> It is probably dependent on which of the two disks actually get used to boot from. >>> >>> So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade. >>> >>> So raid systems need to have their raid arrays fully working before doing an upgrade. >>> >>> At least the above is my interpretation of what I have found. What does everyone else think. >>> >>> >>> Regards, >>> >>> Adolf. >>> >>> >>> On 13/05/2022 15:59, Adolf Belka wrote: >>>> Hi Michael, >>>> >>>> I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules. >>>> >>>> I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot. >>>> >>>> However I did copy the boot directory info for these two cases and found an interesting effect. >>>> >>>> For the problem vm here was the boot directory layout of the CU167 base machine >>>> >>>> drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . >>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. >>>> -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire >>>> drwxr-xr-x 3 root root 16K Jan 1 1970 efi >>>> drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub >>>> -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img >>>> drwx------ 2 root root 16K Sep 30 2021 lost+found >>>> -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire >>>> -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire >>>> >>>> After upgrade and before reboot this had changed to >>>> >>>> drwxr-xr-x 5 root root 4.0K May 13 15:40 . >>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. >>>> -rw-r--r-- 1 root root 179K May 3 04:02 config-5.15.35-ipfire >>>> drwxr-xr-x 3 root root 16K Jan 1 1970 efi >>>> drwxr-xr-x 6 root root 4.0K May 13 15:41 grub >>>> -rw------- 1 root root 26M May 13 15:41 initramfs-5.15.35-ipfire.img >>>> drwx------ 2 root root 16K Sep 30 2021 lost+found >>>> -rw-r--r-- 1 root root 4.7M May 3 04:02 System.map-5.15.35-ipfire >>>> -rw-r--r-- 1 root root 6.7M May 3 04:02 vmlinuz-5.15.35-ipfire >>>> >>>> Dates and times of some of the files/directories have changed as would be expected. >>>> >>>> After Reboot and with the problem the boot directory was now >>>> >>>> drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . >>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. >>>> -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire >>>> drwxr-xr-x 3 root root 4.0K Sep 30 2021 efi >>>> drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub >>>> -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img >>>> drwx------ 2 root root 16K Sep 30 2021 lost+found >>>> -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire >>>> -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire >>>> >>>> Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed >>>> >>>> For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade. >>>> >>>> I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not. >>>> >>>> Regards, >>>> >>>> Adolf. >>>> >>>> On 13/05/2022 12:09, Michael Tremer wrote: >>>>> Hello, >>>>> >>>>>> On 12 May 2022, at 21:10, Adolf Belka adolf.belka@ipfire.org wrote: >>>>>> >>>>>> Hi Michael, >>>>>> >>>>>> On 12/05/2022 14:53, Michael Tremer wrote: >>>>>>> Hello, >>>>>>>> On 12 May 2022, at 12:25, Adolf Belka adolf.belka@ipfire.org wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> On 12/05/2022 11:13, Michael Tremer wrote: >>>>>>>>> Hello, >>>>>>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while. >>>>>>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved. >>>>>>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started? >>>>>>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that. >>>>>>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm. >>>>>>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case? >>>>>>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces. >>>>>>> Okay. That is indeed quite bad then. >>>>>>> Since there is no kernel in 168, is there a chance that we broke the update to 167? >>>>>> I don't know but as far as I have been concerned CU167 has worked well on my vm. >>>>>> Is there something I should look for on my CU167 vm? >>>>>>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface. >>>>>>> Third time lucky isn’t good enough for me :) >>>>>> Me neither :-) >>>>>>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces. >>>>>>>> >>>>>>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers >>>>>>> Have there been any modules loaded? If one loads, they all should load. >>>>>> This is what I get with lsmod >>>>>> >>>>>> Module Size Used by >>>>>> crct10dif_pclmul 16384 1 >>>>>> crc32_pclmul 16384 0 >>>>>> ghash_clmulni_intel 16384 0 >>>>>> serio_raw 20480 0 >>>>>> ohci_pci 20480 0 >>>>>> ata_generic 16384 0 >>>>>> pata_acpi 16384 0 >>>>>> video 57344 0 >>>>> >>>>> It looks like these modules could have come from the initramdisk image. At least that is working well. >>>>> >>>>>> On my running CU1678 vm lsmod gives the following >>>>>> >>>>>> Module Size Used by >>>>>> tun 61440 2 >>>>>> nfnetlink_queue 28672 1 >>>>>> xt_NFQUEUE 16384 8 >>>>>> xt_MASQUERADE 20480 1 >>>>>> cfg80211 1036288 0 >>>>>> rfkill 32768 1 cfg80211 >>>>>> 8021q 40960 0 >>>>>> garp 16384 1 8021q >>>>>> xt_set 16384 260 >>>>>> ip_set_hash_net 49152 258 >>>>>> ip_set 57344 2 xt_set,ip_set_hash_net >>>>>> xt_hashlimit 20480 2 >>>>>> xt_multiport 20480 4 >>>>>> xt_policy 16384 5 >>>>>> xt_TCPMSS 16384 1 >>>>>> xt_conntrack 16384 7 >>>>>> xt_comment 16384 18 >>>>>> ipt_REJECT 16384 1 >>>>>> nf_reject_ipv4 16384 1 ipt_REJECT >>>>>> xt_LOG 20480 26 >>>>>> xt_limit 16384 25 >>>>>> xt_mark 16384 27 >>>>>> xt_connmark 16384 2 >>>>>> nf_log_syslog 24576 26 >>>>>> iptable_raw 16384 0 >>>>>> iptable_mangle 16384 1 >>>>>> iptable_filter 16384 1 >>>>>> vfat 24576 1 >>>>>> fat 90112 1 vfat >>>>>> sch_cake 36864 4 >>>>>> intel_powerclamp 20480 0 >>>>>> psmouse 184320 0 >>>>>> pcspkr 16384 0 >>>>>> i2c_piix4 28672 0 >>>>>> e1000 163840 0 >>>>>> i2c_core 106496 2 psmouse,i2c_piix4 >>>>>> crct10dif_pclmul 16384 1 >>>>>> crc32_pclmul 16384 0 >>>>>> ata_generic 16384 0 >>>>>> pata_acpi 16384 0 >>>>>> ghash_clmulni_intel 16384 0 >>>>>> serio_raw 20480 0 >>>>>> ohci_pci 20480 0 >>>>>> video 57344 0 >>>>> >>>>> Yes, this looks more like it. >>>>> >>>>>> The last 8 entries are the same but a whole load of others are missing. >>>>> >>>>> Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t. >>>>> >>>>> I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else. >>>>> >>>>> -Michael >>>>> >>>>>> >>>>>> Regards, >>>>>> Adolf. >>>>>>>> Tried modprobe e1000 but this came back with the following error >>>>>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service >>>>>>>> >>>>>>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once. >>>>>>>> >>>>>>>> Regards >>>>>>>> Adolf >>>>>>>>>> On 11 May 2022, at 20:08, Adolf Belka adolf.belka@ipfire.org wrote: >>>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> On 11/05/2022 20:48, Jon Murphy wrote: >>>>>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info: >>>>>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* >>>>>>>>>>> Shouldn’t this be Core Update 168? >>>>>>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases. >>>>>>>>>> >>>>>>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm. >>>>>>>>>> >>>>>>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box. >>>>>>>>>> >>>>>>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box. >>>>>>>>>> >>>>>>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box. >>>>>>>>> That seems to be a bug that should not be there. Did we recently update apache? >>>>>>>>> -Michael >>>>>>>>>> >>>>>>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem. >>>>>>>>>> >>>>>>>>>> I think I am going to raise a bug on this. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Adolf. >>>>>>>>>>> Jon >>>>>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi All, >>>>>>>>>>>> >>>>>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote: >>>>>>>>>>>>> Hi Leo, >>>>>>>>>>>>> >>>>>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote: >>>>>>>>>>>>>> Hi Adolf, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code: >>>>>>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that. >>>>>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 >>>>>>>>>>>>> Now I see it in the code. Thanks very much. >>>>>>>>>>>>>> >>>>>>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems! >>>>>>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions. >>>>>>>>>>>>> >>>>>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. >>>>>>>>>>>>> Will wait to hear other inputs on this problem. >>>>>>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again. >>>>>>>>>>>>> >>>>>>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones. >>>>>>>>>>>> >>>>>>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm. >>>>>>>>>>>> >>>>>>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/ >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> >>>>>>>>>>>> Adolf. >>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Adolf. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best regards >>>>>>>>>>>>>> Leo >>>>>>>>>>>>>> >>>>>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> First try:- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> starting >>>>>>>>>>>>>>> DUID 00:04:70:........ >>>>>>>>>>>>>>> red0: interface not found >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Loading of module with unavailable key is rejected >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Second try:- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This time running the update took over 10 mins. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Repository was set back to Stable. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Before running the pakfire directory had the following >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> After running it had >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >>>>>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Rebooted. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Third try:- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >>>>>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! >>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce >>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 >>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. >>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 >>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes >>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. >>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. >>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... >>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 >>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. >>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... >>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 >>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Rebooted >>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>>>>>>>> Several repeats of the following two lines on the console screen:- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) >>>>>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I have given up at this point. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Adolf.
Please note the patch that I just posted to fix any broken RAID arrays.
Does anyone have any systems that we can use to test this?
-Michael
On 17 May 2022, at 09:09, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 13/05/2022 22:37, Michael Tremer wrote:
Hello, Okay, this is really bad then if we are breaking RAID installations. I tried to debug this, but VirtualBox is not my friend today. I asked some bug reporter to provide me with the logs that I want over here: https://bugzilla.ipfire.org/show_bug.cgi?id=12862
After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
Thanks very much for all the help.
Regards, Adolf.
-Michael
On 13 May 2022, at 18:12, Adolf Belka adolf.belka@ipfire.org wrote:
Hi All,
I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
The vm shows the following
-bash-5.1$ cat /proc/mdstat Personalities : md127 : inactive sdb[1](S) 52428724 blocks super 1.0
unused devices: <none>
It looks like IPFire was only running on sda.
So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
The CU168 vm that is working has the same as the above setting.
The CU168 vm that is not working has the following setting
-bash-5.1$ cat /proc/mdstat Personalities : md127 : inactive sda[0](S) 52428724 blocks super 1.0
unused devices: <none>
So it looks like it has booted up on the disk that was not updated from the raid pair.
This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
It is probably dependent on which of the two disks actually get used to boot from.
So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
So raid systems need to have their raid arrays fully working before doing an upgrade.
At least the above is my interpretation of what I have found. What does everyone else think.
Regards,
Adolf.
On 13/05/2022 15:59, Adolf Belka wrote:
Hi Michael,
I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
However I did copy the boot directory info for these two cases and found an interesting effect.
For the problem vm here was the boot directory layout of the CU167 base machine
drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire drwxr-xr-x 3 root root 16K Jan 1 1970 efi drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
After upgrade and before reboot this had changed to
drwxr-xr-x 5 root root 4.0K May 13 15:40 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K May 3 04:02 config-5.15.35-ipfire drwxr-xr-x 3 root root 16K Jan 1 1970 efi drwxr-xr-x 6 root root 4.0K May 13 15:41 grub -rw------- 1 root root 26M May 13 15:41 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M May 3 04:02 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M May 3 04:02 vmlinuz-5.15.35-ipfire
Dates and times of some of the files/directories have changed as would be expected.
After Reboot and with the problem the boot directory was now
drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire drwxr-xr-x 3 root root 4.0K Sep 30 2021 efi drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
Regards,
Adolf.
On 13/05/2022 12:09, Michael Tremer wrote:
Hello,
On 12 May 2022, at 21:10, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 12/05/2022 14:53, Michael Tremer wrote: > Hello, >> On 12 May 2022, at 12:25, Adolf Belka adolf.belka@ipfire.org wrote: >> >> Hi, >> >> On 12/05/2022 11:13, Michael Tremer wrote: >>> Hello, >>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while. >>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved. >>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started? >>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that. >> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm. >>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case? >> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces. > Okay. That is indeed quite bad then. > Since there is no kernel in 168, is there a chance that we broke the update to 167? I don't know but as far as I have been concerned CU167 has worked well on my vm. Is there something I should look for on my CU167 vm? >> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface. > Third time lucky isn’t good enough for me :) Me neither :-) >> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces. >> >> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers > Have there been any modules loaded? If one loads, they all should load. This is what I get with lsmod
Module Size Used by crct10dif_pclmul 16384 1 crc32_pclmul 16384 0 ghash_clmulni_intel 16384 0 serio_raw 20480 0 ohci_pci 20480 0 ata_generic 16384 0 pata_acpi 16384 0 video 57344 0
It looks like these modules could have come from the initramdisk image. At least that is working well.
On my running CU1678 vm lsmod gives the following
Module Size Used by tun 61440 2 nfnetlink_queue 28672 1 xt_NFQUEUE 16384 8 xt_MASQUERADE 20480 1 cfg80211 1036288 0 rfkill 32768 1 cfg80211 8021q 40960 0 garp 16384 1 8021q xt_set 16384 260 ip_set_hash_net 49152 258 ip_set 57344 2 xt_set,ip_set_hash_net xt_hashlimit 20480 2 xt_multiport 20480 4 xt_policy 16384 5 xt_TCPMSS 16384 1 xt_conntrack 16384 7 xt_comment 16384 18 ipt_REJECT 16384 1 nf_reject_ipv4 16384 1 ipt_REJECT xt_LOG 20480 26 xt_limit 16384 25 xt_mark 16384 27 xt_connmark 16384 2 nf_log_syslog 24576 26 iptable_raw 16384 0 iptable_mangle 16384 1 iptable_filter 16384 1 vfat 24576 1 fat 90112 1 vfat sch_cake 36864 4 intel_powerclamp 20480 0 psmouse 184320 0 pcspkr 16384 0 i2c_piix4 28672 0 e1000 163840 0 i2c_core 106496 2 psmouse,i2c_piix4 crct10dif_pclmul 16384 1 crc32_pclmul 16384 0 ata_generic 16384 0 pata_acpi 16384 0 ghash_clmulni_intel 16384 0 serio_raw 20480 0 ohci_pci 20480 0 video 57344 0
Yes, this looks more like it.
The last 8 entries are the same but a whole load of others are missing.
Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
-Michael
Regards, Adolf. >> Tried modprobe e1000 but this came back with the following error >> modprobe: ERROR: could not insert 'e1000': Key was rejected by service >> >> So out of 7 or 8 reboots the vm booted with the network drivers loaded once. >> >> Regards >> Adolf >>>> On 11 May 2022, at 20:08, Adolf Belka adolf.belka@ipfire.org wrote: >>>> >>>> Hi All, >>>> >>>> On 11/05/2022 20:48, Jon Murphy wrote: >>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info: >>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* >>>>> Shouldn’t this be Core Update 168? >>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases. >>>> >>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm. >>>> >>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box. >>>> >>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box. >>>> >>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box. >>> That seems to be a bug that should not be there. Did we recently update apache? >>> -Michael >>>> >>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem. >>>> >>>> I think I am going to raise a bug on this. >>>> >>>> Regards, >>>> Adolf. >>>>> Jon >>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote: >>>>>> >>>>>> Hi All, >>>>>> >>>>>> On 11/05/2022 16:00, Adolf Belka wrote: >>>>>>> Hi Leo, >>>>>>> >>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote: >>>>>>>> Hi Adolf, >>>>>>>> >>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code: >>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that. >>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 >>>>>>> Now I see it in the code. Thanks very much. >>>>>>>> >>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems! >>>>>>> Thanks for your help anyway. It's got rid of one of my questions. >>>>>>> >>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. >>>>>>> Will wait to hear other inputs on this problem. >>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again. >>>>>>> >>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones. >>>>>> >>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm. >>>>>> >>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/ >>>>>> >>>>>> Regards, >>>>>> >>>>>> Adolf. >>>>>> >>>>>>> Regards, >>>>>>> Adolf. >>>>>>> >>>>>>>> >>>>>>>> Best regards >>>>>>>> Leo >>>>>>>> >>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts. >>>>>>>>> >>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case. >>>>>>>>> >>>>>>>>> >>>>>>>>> First try:- >>>>>>>>> >>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had >>>>>>>>> >>>>>>>>> Core Update 167 Development Build: master/c22d834c >>>>>>>>> >>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases. >>>>>>>>> >>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped. >>>>>>>>> >>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>> >>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access. >>>>>>>>> >>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message >>>>>>>>> >>>>>>>>> starting >>>>>>>>> DUID 00:04:70:........ >>>>>>>>> red0: interface not found >>>>>>>>> >>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0. >>>>>>>>> >>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same >>>>>>>>> >>>>>>>>> Loading of module with unavailable key is rejected >>>>>>>>> >>>>>>>>> >>>>>>>>> Second try:- >>>>>>>>> >>>>>>>>> This time running the update took over 10 mins. >>>>>>>>> >>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167. >>>>>>>>> >>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>>>>>>> >>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time. >>>>>>>>> >>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28 >>>>>>>>> >>>>>>>>> Repository was set back to Stable. >>>>>>>>> >>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168. >>>>>>>>> >>>>>>>>> Before running the pakfire directory had the following >>>>>>>>> >>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>> >>>>>>>>> After running it had >>>>>>>>> >>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >>>>>>>>> >>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen. >>>>>>>>> >>>>>>>>> Rebooted. >>>>>>>>> >>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c >>>>>>>>> >>>>>>>>> /opt/pakfire/db/core/mine has 168 in it >>>>>>>>> >>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168. >>>>>>>>> >>>>>>>>> >>>>>>>>> Third try:- >>>>>>>>> >>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>> >>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly. >>>>>>>>> >>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >>>>>>>>> >>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell. >>>>>>>>> >>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! >>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce >>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 >>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire >>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. >>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 >>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes >>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. >>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire >>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. >>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... >>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 >>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... >>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. >>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... >>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 >>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts... >>>>>>>>> >>>>>>>>> >>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. >>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce >>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... >>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. >>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire >>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. >>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... >>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing. >>>>>>>>> >>>>>>>>> Rebooted >>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>> Several repeats of the following two lines on the console screen:- >>>>>>>>> >>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) >>>>>>>>> Perhaps iptables or your kernel needs to be upgraded. >>>>>>>>> >>>>>>>>> >>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present. >>>>>>>>> >>>>>>>>> >>>>>>>>> I have given up at this point. >>>>>>>>> >>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know. >>>>>>>>> >>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Adolf.
Hi Michael,
On 19/05/2022 10:59, Michael Tremer wrote:
Please note the patch that I just posted to fix any broken RAID arrays.
Does anyone have any systems that we can use to test this?
Does the script need to be run manually or has it been added to the nightlies together with the rd.auto patch so that I can do a CU168 Testing update evaluation?
Regards, Adolf.
-Michael
On 17 May 2022, at 09:09, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 13/05/2022 22:37, Michael Tremer wrote:
Hello, Okay, this is really bad then if we are breaking RAID installations. I tried to debug this, but VirtualBox is not my friend today. I asked some bug reporter to provide me with the logs that I want over here: https://bugzilla.ipfire.org/show_bug.cgi?id=12862
After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
Thanks very much for all the help.
Regards, Adolf.
-Michael
On 13 May 2022, at 18:12, Adolf Belka adolf.belka@ipfire.org wrote:
Hi All,
I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
The vm shows the following
-bash-5.1$ cat /proc/mdstat Personalities : md127 : inactive sdb[1](S) 52428724 blocks super 1.0
unused devices: <none>
It looks like IPFire was only running on sda.
So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
The CU168 vm that is working has the same as the above setting.
The CU168 vm that is not working has the following setting
-bash-5.1$ cat /proc/mdstat Personalities : md127 : inactive sda[0](S) 52428724 blocks super 1.0
unused devices: <none>
So it looks like it has booted up on the disk that was not updated from the raid pair.
This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
It is probably dependent on which of the two disks actually get used to boot from.
So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
So raid systems need to have their raid arrays fully working before doing an upgrade.
At least the above is my interpretation of what I have found. What does everyone else think.
Regards,
Adolf.
On 13/05/2022 15:59, Adolf Belka wrote:
Hi Michael,
I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
However I did copy the boot directory info for these two cases and found an interesting effect.
For the problem vm here was the boot directory layout of the CU167 base machine
drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire drwxr-xr-x 3 root root 16K Jan 1 1970 efi drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
After upgrade and before reboot this had changed to
drwxr-xr-x 5 root root 4.0K May 13 15:40 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K May 3 04:02 config-5.15.35-ipfire drwxr-xr-x 3 root root 16K Jan 1 1970 efi drwxr-xr-x 6 root root 4.0K May 13 15:41 grub -rw------- 1 root root 26M May 13 15:41 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M May 3 04:02 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M May 3 04:02 vmlinuz-5.15.35-ipfire
Dates and times of some of the files/directories have changed as would be expected.
After Reboot and with the problem the boot directory was now
drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire drwxr-xr-x 3 root root 4.0K Sep 30 2021 efi drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
Regards,
Adolf.
On 13/05/2022 12:09, Michael Tremer wrote:
Hello,
> On 12 May 2022, at 21:10, Adolf Belka adolf.belka@ipfire.org wrote: > > Hi Michael, > > On 12/05/2022 14:53, Michael Tremer wrote: >> Hello, >>> On 12 May 2022, at 12:25, Adolf Belka adolf.belka@ipfire.org wrote: >>> >>> Hi, >>> >>> On 12/05/2022 11:13, Michael Tremer wrote: >>>> Hello, >>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while. >>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved. >>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started? >>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that. >>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm. >>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case? >>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces. >> Okay. That is indeed quite bad then. >> Since there is no kernel in 168, is there a chance that we broke the update to 167? > I don't know but as far as I have been concerned CU167 has worked well on my vm. > Is there something I should look for on my CU167 vm? >>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface. >> Third time lucky isn’t good enough for me :) > Me neither :-) >>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces. >>> >>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers >> Have there been any modules loaded? If one loads, they all should load. > This is what I get with lsmod > > Module Size Used by > crct10dif_pclmul 16384 1 > crc32_pclmul 16384 0 > ghash_clmulni_intel 16384 0 > serio_raw 20480 0 > ohci_pci 20480 0 > ata_generic 16384 0 > pata_acpi 16384 0 > video 57344 0
It looks like these modules could have come from the initramdisk image. At least that is working well.
> On my running CU1678 vm lsmod gives the following > > Module Size Used by > tun 61440 2 > nfnetlink_queue 28672 1 > xt_NFQUEUE 16384 8 > xt_MASQUERADE 20480 1 > cfg80211 1036288 0 > rfkill 32768 1 cfg80211 > 8021q 40960 0 > garp 16384 1 8021q > xt_set 16384 260 > ip_set_hash_net 49152 258 > ip_set 57344 2 xt_set,ip_set_hash_net > xt_hashlimit 20480 2 > xt_multiport 20480 4 > xt_policy 16384 5 > xt_TCPMSS 16384 1 > xt_conntrack 16384 7 > xt_comment 16384 18 > ipt_REJECT 16384 1 > nf_reject_ipv4 16384 1 ipt_REJECT > xt_LOG 20480 26 > xt_limit 16384 25 > xt_mark 16384 27 > xt_connmark 16384 2 > nf_log_syslog 24576 26 > iptable_raw 16384 0 > iptable_mangle 16384 1 > iptable_filter 16384 1 > vfat 24576 1 > fat 90112 1 vfat > sch_cake 36864 4 > intel_powerclamp 20480 0 > psmouse 184320 0 > pcspkr 16384 0 > i2c_piix4 28672 0 > e1000 163840 0 > i2c_core 106496 2 psmouse,i2c_piix4 > crct10dif_pclmul 16384 1 > crc32_pclmul 16384 0 > ata_generic 16384 0 > pata_acpi 16384 0 > ghash_clmulni_intel 16384 0 > serio_raw 20480 0 > ohci_pci 20480 0 > video 57344 0
Yes, this looks more like it.
> The last 8 entries are the same but a whole load of others are missing.
Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
-Michael
> > Regards, > Adolf. >>> Tried modprobe e1000 but this came back with the following error >>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service >>> >>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once. >>> >>> Regards >>> Adolf >>>>> On 11 May 2022, at 20:08, Adolf Belka adolf.belka@ipfire.org wrote: >>>>> >>>>> Hi All, >>>>> >>>>> On 11/05/2022 20:48, Jon Murphy wrote: >>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info: >>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* >>>>>> Shouldn’t this be Core Update 168? >>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases. >>>>> >>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm. >>>>> >>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box. >>>>> >>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box. >>>>> >>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box. >>>> That seems to be a bug that should not be there. Did we recently update apache? >>>> -Michael >>>>> >>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem. >>>>> >>>>> I think I am going to raise a bug on this. >>>>> >>>>> Regards, >>>>> Adolf. >>>>>> Jon >>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote: >>>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> On 11/05/2022 16:00, Adolf Belka wrote: >>>>>>>> Hi Leo, >>>>>>>> >>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote: >>>>>>>>> Hi Adolf, >>>>>>>>> >>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code: >>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that. >>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 >>>>>>>> Now I see it in the code. Thanks very much. >>>>>>>>> >>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems! >>>>>>>> Thanks for your help anyway. It's got rid of one of my questions. >>>>>>>> >>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. >>>>>>>> Will wait to hear other inputs on this problem. >>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again. >>>>>>>> >>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones. >>>>>>> >>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm. >>>>>>> >>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/ >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Adolf. >>>>>>> >>>>>>>> Regards, >>>>>>>> Adolf. >>>>>>>> >>>>>>>>> >>>>>>>>> Best regards >>>>>>>>> Leo >>>>>>>>> >>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts. >>>>>>>>>> >>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> First try:- >>>>>>>>>> >>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had >>>>>>>>>> >>>>>>>>>> Core Update 167 Development Build: master/c22d834c >>>>>>>>>> >>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases. >>>>>>>>>> >>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped. >>>>>>>>>> >>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>>> >>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access. >>>>>>>>>> >>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message >>>>>>>>>> >>>>>>>>>> starting >>>>>>>>>> DUID 00:04:70:........ >>>>>>>>>> red0: interface not found >>>>>>>>>> >>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0. >>>>>>>>>> >>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same >>>>>>>>>> >>>>>>>>>> Loading of module with unavailable key is rejected >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Second try:- >>>>>>>>>> >>>>>>>>>> This time running the update took over 10 mins. >>>>>>>>>> >>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167. >>>>>>>>>> >>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>>>>>>>> >>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time. >>>>>>>>>> >>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28 >>>>>>>>>> >>>>>>>>>> Repository was set back to Stable. >>>>>>>>>> >>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168. >>>>>>>>>> >>>>>>>>>> Before running the pakfire directory had the following >>>>>>>>>> >>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>>> >>>>>>>>>> After running it had >>>>>>>>>> >>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >>>>>>>>>> >>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen. >>>>>>>>>> >>>>>>>>>> Rebooted. >>>>>>>>>> >>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c >>>>>>>>>> >>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it >>>>>>>>>> >>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Third try:- >>>>>>>>>> >>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>>> >>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly. >>>>>>>>>> >>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >>>>>>>>>> >>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell. >>>>>>>>>> >>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! >>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce >>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 >>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire >>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. >>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 >>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes >>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. >>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire >>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. >>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... >>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 >>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... >>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. >>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... >>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 >>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts... >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. >>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce >>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... >>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. >>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire >>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. >>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... >>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing. >>>>>>>>>> >>>>>>>>>> Rebooted >>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>>> Several repeats of the following two lines on the console screen:- >>>>>>>>>> >>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) >>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I have given up at this point. >>>>>>>>>> >>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know. >>>>>>>>>> >>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Adolf.
Hello,
On 19 May 2022, at 11:57, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 19/05/2022 10:59, Michael Tremer wrote:
Please note the patch that I just posted to fix any broken RAID arrays. Does anyone have any systems that we can use to test this?
Does the script need to be run manually or has it been added to the nightlies together with the rd.auto patch so that I can do a CU168 Testing update evaluation?
My patch includes that it will run automatically with the updater. However, that is not part of the build yet, because I would like it to be confirmed that I do not destroy anything. The whole script is a rather risky operation :)
Regards, Adolf.
-Michael
On 17 May 2022, at 09:09, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 13/05/2022 22:37, Michael Tremer wrote:
Hello, Okay, this is really bad then if we are breaking RAID installations. I tried to debug this, but VirtualBox is not my friend today. I asked some bug reporter to provide me with the logs that I want over here: https://bugzilla.ipfire.org/show_bug.cgi?id=12862
After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
Thanks very much for all the help.
Regards, Adolf.
-Michael
On 13 May 2022, at 18:12, Adolf Belka adolf.belka@ipfire.org wrote:
Hi All,
I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
The vm shows the following
-bash-5.1$ cat /proc/mdstat Personalities : md127 : inactive sdb[1](S) 52428724 blocks super 1.0
unused devices: <none>
It looks like IPFire was only running on sda.
So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
The CU168 vm that is working has the same as the above setting.
The CU168 vm that is not working has the following setting
-bash-5.1$ cat /proc/mdstat Personalities : md127 : inactive sda[0](S) 52428724 blocks super 1.0
unused devices: <none>
So it looks like it has booted up on the disk that was not updated from the raid pair.
This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
It is probably dependent on which of the two disks actually get used to boot from.
So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
So raid systems need to have their raid arrays fully working before doing an upgrade.
At least the above is my interpretation of what I have found. What does everyone else think.
Regards,
Adolf.
On 13/05/2022 15:59, Adolf Belka wrote:
Hi Michael,
I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
However I did copy the boot directory info for these two cases and found an interesting effect.
For the problem vm here was the boot directory layout of the CU167 base machine
drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire drwxr-xr-x 3 root root 16K Jan 1 1970 efi drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
After upgrade and before reboot this had changed to
drwxr-xr-x 5 root root 4.0K May 13 15:40 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K May 3 04:02 config-5.15.35-ipfire drwxr-xr-x 3 root root 16K Jan 1 1970 efi drwxr-xr-x 6 root root 4.0K May 13 15:41 grub -rw------- 1 root root 26M May 13 15:41 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M May 3 04:02 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M May 3 04:02 vmlinuz-5.15.35-ipfire
Dates and times of some of the files/directories have changed as would be expected.
After Reboot and with the problem the boot directory was now
drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire drwxr-xr-x 3 root root 4.0K Sep 30 2021 efi drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img drwx------ 2 root root 16K Sep 30 2021 lost+found -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
Regards,
Adolf.
On 13/05/2022 12:09, Michael Tremer wrote: > Hello, > >> On 12 May 2022, at 21:10, Adolf Belka adolf.belka@ipfire.org wrote: >> >> Hi Michael, >> >> On 12/05/2022 14:53, Michael Tremer wrote: >>> Hello, >>>> On 12 May 2022, at 12:25, Adolf Belka adolf.belka@ipfire.org wrote: >>>> >>>> Hi, >>>> >>>> On 12/05/2022 11:13, Michael Tremer wrote: >>>>> Hello, >>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while. >>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved. >>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started? >>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that. >>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm. >>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case? >>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces. >>> Okay. That is indeed quite bad then. >>> Since there is no kernel in 168, is there a chance that we broke the update to 167? >> I don't know but as far as I have been concerned CU167 has worked well on my vm. >> Is there something I should look for on my CU167 vm? >>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface. >>> Third time lucky isn’t good enough for me :) >> Me neither :-) >>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces. >>>> >>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers >>> Have there been any modules loaded? If one loads, they all should load. >> This is what I get with lsmod >> >> Module Size Used by >> crct10dif_pclmul 16384 1 >> crc32_pclmul 16384 0 >> ghash_clmulni_intel 16384 0 >> serio_raw 20480 0 >> ohci_pci 20480 0 >> ata_generic 16384 0 >> pata_acpi 16384 0 >> video 57344 0 > > It looks like these modules could have come from the initramdisk image. At least that is working well. > >> On my running CU1678 vm lsmod gives the following >> >> Module Size Used by >> tun 61440 2 >> nfnetlink_queue 28672 1 >> xt_NFQUEUE 16384 8 >> xt_MASQUERADE 20480 1 >> cfg80211 1036288 0 >> rfkill 32768 1 cfg80211 >> 8021q 40960 0 >> garp 16384 1 8021q >> xt_set 16384 260 >> ip_set_hash_net 49152 258 >> ip_set 57344 2 xt_set,ip_set_hash_net >> xt_hashlimit 20480 2 >> xt_multiport 20480 4 >> xt_policy 16384 5 >> xt_TCPMSS 16384 1 >> xt_conntrack 16384 7 >> xt_comment 16384 18 >> ipt_REJECT 16384 1 >> nf_reject_ipv4 16384 1 ipt_REJECT >> xt_LOG 20480 26 >> xt_limit 16384 25 >> xt_mark 16384 27 >> xt_connmark 16384 2 >> nf_log_syslog 24576 26 >> iptable_raw 16384 0 >> iptable_mangle 16384 1 >> iptable_filter 16384 1 >> vfat 24576 1 >> fat 90112 1 vfat >> sch_cake 36864 4 >> intel_powerclamp 20480 0 >> psmouse 184320 0 >> pcspkr 16384 0 >> i2c_piix4 28672 0 >> e1000 163840 0 >> i2c_core 106496 2 psmouse,i2c_piix4 >> crct10dif_pclmul 16384 1 >> crc32_pclmul 16384 0 >> ata_generic 16384 0 >> pata_acpi 16384 0 >> ghash_clmulni_intel 16384 0 >> serio_raw 20480 0 >> ohci_pci 20480 0 >> video 57344 0 > > Yes, this looks more like it. > >> The last 8 entries are the same but a whole load of others are missing. > > Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t. > > I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else. > > -Michael > >> >> Regards, >> Adolf. >>>> Tried modprobe e1000 but this came back with the following error >>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service >>>> >>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once. >>>> >>>> Regards >>>> Adolf >>>>>> On 11 May 2022, at 20:08, Adolf Belka adolf.belka@ipfire.org wrote: >>>>>> >>>>>> Hi All, >>>>>> >>>>>> On 11/05/2022 20:48, Jon Murphy wrote: >>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info: >>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* >>>>>>> Shouldn’t this be Core Update 168? >>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases. >>>>>> >>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm. >>>>>> >>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box. >>>>>> >>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box. >>>>>> >>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box. >>>>> That seems to be a bug that should not be there. Did we recently update apache? >>>>> -Michael >>>>>> >>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem. >>>>>> >>>>>> I think I am going to raise a bug on this. >>>>>> >>>>>> Regards, >>>>>> Adolf. >>>>>>> Jon >>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote: >>>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote: >>>>>>>>> Hi Leo, >>>>>>>>> >>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote: >>>>>>>>>> Hi Adolf, >>>>>>>>>> >>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code: >>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that. >>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 >>>>>>>>> Now I see it in the code. Thanks very much. >>>>>>>>>> >>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems! >>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions. >>>>>>>>> >>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. >>>>>>>>> Will wait to hear other inputs on this problem. >>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again. >>>>>>>>> >>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones. >>>>>>>> >>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm. >>>>>>>> >>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/ >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Adolf. >>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Adolf. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Best regards >>>>>>>>>> Leo >>>>>>>>>> >>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>>>>>>>>> Hi All, >>>>>>>>>>> >>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts. >>>>>>>>>>> >>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> First try:- >>>>>>>>>>> >>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had >>>>>>>>>>> >>>>>>>>>>> Core Update 167 Development Build: master/c22d834c >>>>>>>>>>> >>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases. >>>>>>>>>>> >>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped. >>>>>>>>>>> >>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>>>> >>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access. >>>>>>>>>>> >>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message >>>>>>>>>>> >>>>>>>>>>> starting >>>>>>>>>>> DUID 00:04:70:........ >>>>>>>>>>> red0: interface not found >>>>>>>>>>> >>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0. >>>>>>>>>>> >>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same >>>>>>>>>>> >>>>>>>>>>> Loading of module with unavailable key is rejected >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Second try:- >>>>>>>>>>> >>>>>>>>>>> This time running the update took over 10 mins. >>>>>>>>>>> >>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167. >>>>>>>>>>> >>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>>>>>>>>> >>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time. >>>>>>>>>>> >>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28 >>>>>>>>>>> >>>>>>>>>>> Repository was set back to Stable. >>>>>>>>>>> >>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168. >>>>>>>>>>> >>>>>>>>>>> Before running the pakfire directory had the following >>>>>>>>>>> >>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>>>> >>>>>>>>>>> After running it had >>>>>>>>>>> >>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >>>>>>>>>>> >>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen. >>>>>>>>>>> >>>>>>>>>>> Rebooted. >>>>>>>>>>> >>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c >>>>>>>>>>> >>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it >>>>>>>>>>> >>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Third try:- >>>>>>>>>>> >>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>>>> >>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly. >>>>>>>>>>> >>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >>>>>>>>>>> >>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell. >>>>>>>>>>> >>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! >>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce >>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 >>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. >>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 >>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes >>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. >>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. >>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... >>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 >>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... >>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. >>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... >>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 >>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts... >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... >>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing. >>>>>>>>>>> >>>>>>>>>>> Rebooted >>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>>>> Several repeats of the following two lines on the console screen:- >>>>>>>>>>> >>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) >>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I have given up at this point. >>>>>>>>>>> >>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know. >>>>>>>>>>> >>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> Adolf. >
Hi,
On 19/05/2022 13:14, Michael Tremer wrote:
Hello,
On 19 May 2022, at 11:57, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 19/05/2022 10:59, Michael Tremer wrote:
Please note the patch that I just posted to fix any broken RAID arrays. Does anyone have any systems that we can use to test this?
Does the script need to be run manually or has it been added to the nightlies together with the rd.auto patch so that I can do a CU168 Testing update evaluation?
My patch includes that it will run automatically with the updater. However, that is not part of the build yet, because I would like it to be confirmed that I do not destroy anything. The whole script is a rather risky operation :)
Okay, clear. Then I will execute the rd.auto change reboot and then run the script.
Will feedback how things go. Adolf.
Regards, Adolf.
-Michael
On 17 May 2022, at 09:09, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 13/05/2022 22:37, Michael Tremer wrote:
Hello, Okay, this is really bad then if we are breaking RAID installations. I tried to debug this, but VirtualBox is not my friend today. I asked some bug reporter to provide me with the logs that I want over here: https://bugzilla.ipfire.org/show_bug.cgi?id=12862
After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
Thanks very much for all the help.
Regards, Adolf.
-Michael
On 13 May 2022, at 18:12, Adolf Belka adolf.belka@ipfire.org wrote:
Hi All,
I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
The vm shows the following
-bash-5.1$ cat /proc/mdstat Personalities : md127 : inactive sdb[1](S) 52428724 blocks super 1.0
unused devices: <none>
It looks like IPFire was only running on sda.
So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
The CU168 vm that is working has the same as the above setting.
The CU168 vm that is not working has the following setting
-bash-5.1$ cat /proc/mdstat Personalities : md127 : inactive sda[0](S) 52428724 blocks super 1.0
unused devices: <none>
So it looks like it has booted up on the disk that was not updated from the raid pair.
This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
It is probably dependent on which of the two disks actually get used to boot from.
So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
So raid systems need to have their raid arrays fully working before doing an upgrade.
At least the above is my interpretation of what I have found. What does everyone else think.
Regards,
Adolf.
On 13/05/2022 15:59, Adolf Belka wrote: > Hi Michael, > > I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules. > > I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot. > > However I did copy the boot directory info for these two cases and found an interesting effect. > > For the problem vm here was the boot directory layout of the CU167 base machine > > drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . > drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. > -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire > drwxr-xr-x 3 root root 16K Jan 1 1970 efi > drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub > -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img > drwx------ 2 root root 16K Sep 30 2021 lost+found > -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire > -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire > > After upgrade and before reboot this had changed to > > drwxr-xr-x 5 root root 4.0K May 13 15:40 . > drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. > -rw-r--r-- 1 root root 179K May 3 04:02 config-5.15.35-ipfire > drwxr-xr-x 3 root root 16K Jan 1 1970 efi > drwxr-xr-x 6 root root 4.0K May 13 15:41 grub > -rw------- 1 root root 26M May 13 15:41 initramfs-5.15.35-ipfire.img > drwx------ 2 root root 16K Sep 30 2021 lost+found > -rw-r--r-- 1 root root 4.7M May 3 04:02 System.map-5.15.35-ipfire > -rw-r--r-- 1 root root 6.7M May 3 04:02 vmlinuz-5.15.35-ipfire > > Dates and times of some of the files/directories have changed as would be expected. > > After Reboot and with the problem the boot directory was now > > drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . > drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. > -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire > drwxr-xr-x 3 root root 4.0K Sep 30 2021 efi > drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub > -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img > drwx------ 2 root root 16K Sep 30 2021 lost+found > -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire > -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire > > Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed > > For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade. > > I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not. > > Regards, > > Adolf. > > On 13/05/2022 12:09, Michael Tremer wrote: >> Hello, >> >>> On 12 May 2022, at 21:10, Adolf Belka adolf.belka@ipfire.org wrote: >>> >>> Hi Michael, >>> >>> On 12/05/2022 14:53, Michael Tremer wrote: >>>> Hello, >>>>> On 12 May 2022, at 12:25, Adolf Belka adolf.belka@ipfire.org wrote: >>>>> >>>>> Hi, >>>>> >>>>> On 12/05/2022 11:13, Michael Tremer wrote: >>>>>> Hello, >>>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while. >>>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved. >>>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started? >>>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that. >>>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm. >>>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case? >>>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces. >>>> Okay. That is indeed quite bad then. >>>> Since there is no kernel in 168, is there a chance that we broke the update to 167? >>> I don't know but as far as I have been concerned CU167 has worked well on my vm. >>> Is there something I should look for on my CU167 vm? >>>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface. >>>> Third time lucky isn’t good enough for me :) >>> Me neither :-) >>>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces. >>>>> >>>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers >>>> Have there been any modules loaded? If one loads, they all should load. >>> This is what I get with lsmod >>> >>> Module Size Used by >>> crct10dif_pclmul 16384 1 >>> crc32_pclmul 16384 0 >>> ghash_clmulni_intel 16384 0 >>> serio_raw 20480 0 >>> ohci_pci 20480 0 >>> ata_generic 16384 0 >>> pata_acpi 16384 0 >>> video 57344 0 >> >> It looks like these modules could have come from the initramdisk image. At least that is working well. >> >>> On my running CU1678 vm lsmod gives the following >>> >>> Module Size Used by >>> tun 61440 2 >>> nfnetlink_queue 28672 1 >>> xt_NFQUEUE 16384 8 >>> xt_MASQUERADE 20480 1 >>> cfg80211 1036288 0 >>> rfkill 32768 1 cfg80211 >>> 8021q 40960 0 >>> garp 16384 1 8021q >>> xt_set 16384 260 >>> ip_set_hash_net 49152 258 >>> ip_set 57344 2 xt_set,ip_set_hash_net >>> xt_hashlimit 20480 2 >>> xt_multiport 20480 4 >>> xt_policy 16384 5 >>> xt_TCPMSS 16384 1 >>> xt_conntrack 16384 7 >>> xt_comment 16384 18 >>> ipt_REJECT 16384 1 >>> nf_reject_ipv4 16384 1 ipt_REJECT >>> xt_LOG 20480 26 >>> xt_limit 16384 25 >>> xt_mark 16384 27 >>> xt_connmark 16384 2 >>> nf_log_syslog 24576 26 >>> iptable_raw 16384 0 >>> iptable_mangle 16384 1 >>> iptable_filter 16384 1 >>> vfat 24576 1 >>> fat 90112 1 vfat >>> sch_cake 36864 4 >>> intel_powerclamp 20480 0 >>> psmouse 184320 0 >>> pcspkr 16384 0 >>> i2c_piix4 28672 0 >>> e1000 163840 0 >>> i2c_core 106496 2 psmouse,i2c_piix4 >>> crct10dif_pclmul 16384 1 >>> crc32_pclmul 16384 0 >>> ata_generic 16384 0 >>> pata_acpi 16384 0 >>> ghash_clmulni_intel 16384 0 >>> serio_raw 20480 0 >>> ohci_pci 20480 0 >>> video 57344 0 >> >> Yes, this looks more like it. >> >>> The last 8 entries are the same but a whole load of others are missing. >> >> Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t. >> >> I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else. >> >> -Michael >> >>> >>> Regards, >>> Adolf. >>>>> Tried modprobe e1000 but this came back with the following error >>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service >>>>> >>>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once. >>>>> >>>>> Regards >>>>> Adolf >>>>>>> On 11 May 2022, at 20:08, Adolf Belka adolf.belka@ipfire.org wrote: >>>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> On 11/05/2022 20:48, Jon Murphy wrote: >>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info: >>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* >>>>>>>> Shouldn’t this be Core Update 168? >>>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases. >>>>>>> >>>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm. >>>>>>> >>>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box. >>>>>>> >>>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box. >>>>>>> >>>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box. >>>>>> That seems to be a bug that should not be there. Did we recently update apache? >>>>>> -Michael >>>>>>> >>>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem. >>>>>>> >>>>>>> I think I am going to raise a bug on this. >>>>>>> >>>>>>> Regards, >>>>>>> Adolf. >>>>>>>> Jon >>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote: >>>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote: >>>>>>>>>> Hi Leo, >>>>>>>>>> >>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote: >>>>>>>>>>> Hi Adolf, >>>>>>>>>>> >>>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code: >>>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that. >>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 >>>>>>>>>> Now I see it in the code. Thanks very much. >>>>>>>>>>> >>>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems! >>>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions. >>>>>>>>>> >>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. >>>>>>>>>> Will wait to hear other inputs on this problem. >>>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again. >>>>>>>>>> >>>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones. >>>>>>>>> >>>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm. >>>>>>>>> >>>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/ >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Adolf. >>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Adolf. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Best regards >>>>>>>>>>> Leo >>>>>>>>>>> >>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>>>>>>>>>> Hi All, >>>>>>>>>>>> >>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts. >>>>>>>>>>>> >>>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> First try:- >>>>>>>>>>>> >>>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had >>>>>>>>>>>> >>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c >>>>>>>>>>>> >>>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases. >>>>>>>>>>>> >>>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped. >>>>>>>>>>>> >>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>>>>> >>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access. >>>>>>>>>>>> >>>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message >>>>>>>>>>>> >>>>>>>>>>>> starting >>>>>>>>>>>> DUID 00:04:70:........ >>>>>>>>>>>> red0: interface not found >>>>>>>>>>>> >>>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0. >>>>>>>>>>>> >>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same >>>>>>>>>>>> >>>>>>>>>>>> Loading of module with unavailable key is rejected >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Second try:- >>>>>>>>>>>> >>>>>>>>>>>> This time running the update took over 10 mins. >>>>>>>>>>>> >>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167. >>>>>>>>>>>> >>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>>>>>>>>>> >>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time. >>>>>>>>>>>> >>>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28 >>>>>>>>>>>> >>>>>>>>>>>> Repository was set back to Stable. >>>>>>>>>>>> >>>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168. >>>>>>>>>>>> >>>>>>>>>>>> Before running the pakfire directory had the following >>>>>>>>>>>> >>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>>>>> >>>>>>>>>>>> After running it had >>>>>>>>>>>> >>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >>>>>>>>>>>> >>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen. >>>>>>>>>>>> >>>>>>>>>>>> Rebooted. >>>>>>>>>>>> >>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c >>>>>>>>>>>> >>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it >>>>>>>>>>>> >>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Third try:- >>>>>>>>>>>> >>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>>>>> >>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly. >>>>>>>>>>>> >>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >>>>>>>>>>>> >>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell. >>>>>>>>>>>> >>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! >>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce >>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 >>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. >>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 >>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes >>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. >>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. >>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... >>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 >>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. >>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... >>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 >>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing. >>>>>>>>>>>> >>>>>>>>>>>> Rebooted >>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>>>>> Several repeats of the following two lines on the console screen:- >>>>>>>>>>>> >>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) >>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I have given up at this point. >>>>>>>>>>>> >>>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know. >>>>>>>>>>>> >>>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> >>>>>>>>>>>> Adolf. >>
Hello,
That would be brilliant. Let me know how it is going.
You would need to run this first:
https://patchwork.ipfire.org/project/ipfire/patch/20220519085634.197389-1-mi...
And then just run the script and reboot. The boot process might stall for a minute. Just let it do its job. And the RAID should come up just fine.
The script will also avoid the previous issues with the file system that you have outlined after an unsuccessful resync.
-Michael
On 19 May 2022, at 12:21, Adolf Belka adolf.belka@ipfire.org wrote:
Hi,
On 19/05/2022 13:14, Michael Tremer wrote:
Hello,
On 19 May 2022, at 11:57, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 19/05/2022 10:59, Michael Tremer wrote:
Please note the patch that I just posted to fix any broken RAID arrays. Does anyone have any systems that we can use to test this?
Does the script need to be run manually or has it been added to the nightlies together with the rd.auto patch so that I can do a CU168 Testing update evaluation?
My patch includes that it will run automatically with the updater. However, that is not part of the build yet, because I would like it to be confirmed that I do not destroy anything. The whole script is a rather risky operation :)
Okay, clear. Then I will execute the rd.auto change reboot and then run the script.
Will feedback how things go. Adolf.
Regards, Adolf.
-Michael
On 17 May 2022, at 09:09, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 13/05/2022 22:37, Michael Tremer wrote:
Hello, Okay, this is really bad then if we are breaking RAID installations. I tried to debug this, but VirtualBox is not my friend today. I asked some bug reporter to provide me with the logs that I want over here: https://bugzilla.ipfire.org/show_bug.cgi?id=12862
After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
Thanks very much for all the help.
Regards, Adolf.
-Michael > On 13 May 2022, at 18:12, Adolf Belka adolf.belka@ipfire.org wrote: > > Hi All, > > I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go. > > I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk. > > I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array. > > The vm shows the following > > -bash-5.1$ cat /proc/mdstat > Personalities : > md127 : inactive sdb[1](S) > 52428724 blocks super 1.0 > > unused devices: <none> > > > It looks like IPFire was only running on sda. > > So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated. > > The CU168 vm that is working has the same as the above setting. > > The CU168 vm that is not working has the following setting > > -bash-5.1$ cat /proc/mdstat > Personalities : > md127 : inactive sda[0](S) > 52428724 blocks super 1.0 > > unused devices: <none> > > So it looks like it has booted up on the disk that was not updated from the raid pair. > > > This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode. > > It is probably dependent on which of the two disks actually get used to boot from. > > So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade. > > So raid systems need to have their raid arrays fully working before doing an upgrade. > > At least the above is my interpretation of what I have found. What does everyone else think. > > > Regards, > > Adolf. > > > On 13/05/2022 15:59, Adolf Belka wrote: >> Hi Michael, >> >> I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules. >> >> I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot. >> >> However I did copy the boot directory info for these two cases and found an interesting effect. >> >> For the problem vm here was the boot directory layout of the CU167 base machine >> >> drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . >> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. >> -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire >> drwxr-xr-x 3 root root 16K Jan 1 1970 efi >> drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub >> -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img >> drwx------ 2 root root 16K Sep 30 2021 lost+found >> -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire >> -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire >> >> After upgrade and before reboot this had changed to >> >> drwxr-xr-x 5 root root 4.0K May 13 15:40 . >> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. >> -rw-r--r-- 1 root root 179K May 3 04:02 config-5.15.35-ipfire >> drwxr-xr-x 3 root root 16K Jan 1 1970 efi >> drwxr-xr-x 6 root root 4.0K May 13 15:41 grub >> -rw------- 1 root root 26M May 13 15:41 initramfs-5.15.35-ipfire.img >> drwx------ 2 root root 16K Sep 30 2021 lost+found >> -rw-r--r-- 1 root root 4.7M May 3 04:02 System.map-5.15.35-ipfire >> -rw-r--r-- 1 root root 6.7M May 3 04:02 vmlinuz-5.15.35-ipfire >> >> Dates and times of some of the files/directories have changed as would be expected. >> >> After Reboot and with the problem the boot directory was now >> >> drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . >> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. >> -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire >> drwxr-xr-x 3 root root 4.0K Sep 30 2021 efi >> drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub >> -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img >> drwx------ 2 root root 16K Sep 30 2021 lost+found >> -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire >> -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire >> >> Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed >> >> For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade. >> >> I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not. >> >> Regards, >> >> Adolf. >> >> On 13/05/2022 12:09, Michael Tremer wrote: >>> Hello, >>> >>>> On 12 May 2022, at 21:10, Adolf Belka adolf.belka@ipfire.org wrote: >>>> >>>> Hi Michael, >>>> >>>> On 12/05/2022 14:53, Michael Tremer wrote: >>>>> Hello, >>>>>> On 12 May 2022, at 12:25, Adolf Belka adolf.belka@ipfire.org wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> On 12/05/2022 11:13, Michael Tremer wrote: >>>>>>> Hello, >>>>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while. >>>>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved. >>>>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started? >>>>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that. >>>>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm. >>>>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case? >>>>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces. >>>>> Okay. That is indeed quite bad then. >>>>> Since there is no kernel in 168, is there a chance that we broke the update to 167? >>>> I don't know but as far as I have been concerned CU167 has worked well on my vm. >>>> Is there something I should look for on my CU167 vm? >>>>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface. >>>>> Third time lucky isn’t good enough for me :) >>>> Me neither :-) >>>>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces. >>>>>> >>>>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers >>>>> Have there been any modules loaded? If one loads, they all should load. >>>> This is what I get with lsmod >>>> >>>> Module Size Used by >>>> crct10dif_pclmul 16384 1 >>>> crc32_pclmul 16384 0 >>>> ghash_clmulni_intel 16384 0 >>>> serio_raw 20480 0 >>>> ohci_pci 20480 0 >>>> ata_generic 16384 0 >>>> pata_acpi 16384 0 >>>> video 57344 0 >>> >>> It looks like these modules could have come from the initramdisk image. At least that is working well. >>> >>>> On my running CU1678 vm lsmod gives the following >>>> >>>> Module Size Used by >>>> tun 61440 2 >>>> nfnetlink_queue 28672 1 >>>> xt_NFQUEUE 16384 8 >>>> xt_MASQUERADE 20480 1 >>>> cfg80211 1036288 0 >>>> rfkill 32768 1 cfg80211 >>>> 8021q 40960 0 >>>> garp 16384 1 8021q >>>> xt_set 16384 260 >>>> ip_set_hash_net 49152 258 >>>> ip_set 57344 2 xt_set,ip_set_hash_net >>>> xt_hashlimit 20480 2 >>>> xt_multiport 20480 4 >>>> xt_policy 16384 5 >>>> xt_TCPMSS 16384 1 >>>> xt_conntrack 16384 7 >>>> xt_comment 16384 18 >>>> ipt_REJECT 16384 1 >>>> nf_reject_ipv4 16384 1 ipt_REJECT >>>> xt_LOG 20480 26 >>>> xt_limit 16384 25 >>>> xt_mark 16384 27 >>>> xt_connmark 16384 2 >>>> nf_log_syslog 24576 26 >>>> iptable_raw 16384 0 >>>> iptable_mangle 16384 1 >>>> iptable_filter 16384 1 >>>> vfat 24576 1 >>>> fat 90112 1 vfat >>>> sch_cake 36864 4 >>>> intel_powerclamp 20480 0 >>>> psmouse 184320 0 >>>> pcspkr 16384 0 >>>> i2c_piix4 28672 0 >>>> e1000 163840 0 >>>> i2c_core 106496 2 psmouse,i2c_piix4 >>>> crct10dif_pclmul 16384 1 >>>> crc32_pclmul 16384 0 >>>> ata_generic 16384 0 >>>> pata_acpi 16384 0 >>>> ghash_clmulni_intel 16384 0 >>>> serio_raw 20480 0 >>>> ohci_pci 20480 0 >>>> video 57344 0 >>> >>> Yes, this looks more like it. >>> >>>> The last 8 entries are the same but a whole load of others are missing. >>> >>> Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t. >>> >>> I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else. >>> >>> -Michael >>> >>>> >>>> Regards, >>>> Adolf. >>>>>> Tried modprobe e1000 but this came back with the following error >>>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service >>>>>> >>>>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once. >>>>>> >>>>>> Regards >>>>>> Adolf >>>>>>>> On 11 May 2022, at 20:08, Adolf Belka adolf.belka@ipfire.org wrote: >>>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> On 11/05/2022 20:48, Jon Murphy wrote: >>>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info: >>>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* >>>>>>>>> Shouldn’t this be Core Update 168? >>>>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases. >>>>>>>> >>>>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm. >>>>>>>> >>>>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box. >>>>>>>> >>>>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box. >>>>>>>> >>>>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box. >>>>>>> That seems to be a bug that should not be there. Did we recently update apache? >>>>>>> -Michael >>>>>>>> >>>>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem. >>>>>>>> >>>>>>>> I think I am going to raise a bug on this. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Adolf. >>>>>>>>> Jon >>>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote: >>>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote: >>>>>>>>>>> Hi Leo, >>>>>>>>>>> >>>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote: >>>>>>>>>>>> Hi Adolf, >>>>>>>>>>>> >>>>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code: >>>>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that. >>>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 >>>>>>>>>>> Now I see it in the code. Thanks very much. >>>>>>>>>>>> >>>>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems! >>>>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions. >>>>>>>>>>> >>>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. >>>>>>>>>>> Will wait to hear other inputs on this problem. >>>>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again. >>>>>>>>>>> >>>>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones. >>>>>>>>>> >>>>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm. >>>>>>>>>> >>>>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/ >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Adolf. >>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Adolf. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Best regards >>>>>>>>>>>> Leo >>>>>>>>>>>> >>>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>>>>>>>>>>> Hi All, >>>>>>>>>>>>> >>>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts. >>>>>>>>>>>>> >>>>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> First try:- >>>>>>>>>>>>> >>>>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had >>>>>>>>>>>>> >>>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c >>>>>>>>>>>>> >>>>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases. >>>>>>>>>>>>> >>>>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped. >>>>>>>>>>>>> >>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>>>>>> >>>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access. >>>>>>>>>>>>> >>>>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message >>>>>>>>>>>>> >>>>>>>>>>>>> starting >>>>>>>>>>>>> DUID 00:04:70:........ >>>>>>>>>>>>> red0: interface not found >>>>>>>>>>>>> >>>>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0. >>>>>>>>>>>>> >>>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same >>>>>>>>>>>>> >>>>>>>>>>>>> Loading of module with unavailable key is rejected >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Second try:- >>>>>>>>>>>>> >>>>>>>>>>>>> This time running the update took over 10 mins. >>>>>>>>>>>>> >>>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167. >>>>>>>>>>>>> >>>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>>>>>>>>>>> >>>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time. >>>>>>>>>>>>> >>>>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28 >>>>>>>>>>>>> >>>>>>>>>>>>> Repository was set back to Stable. >>>>>>>>>>>>> >>>>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168. >>>>>>>>>>>>> >>>>>>>>>>>>> Before running the pakfire directory had the following >>>>>>>>>>>>> >>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>>>>>> >>>>>>>>>>>>> After running it had >>>>>>>>>>>>> >>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >>>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >>>>>>>>>>>>> >>>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen. >>>>>>>>>>>>> >>>>>>>>>>>>> Rebooted. >>>>>>>>>>>>> >>>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c >>>>>>>>>>>>> >>>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it >>>>>>>>>>>>> >>>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Third try:- >>>>>>>>>>>>> >>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>>>>>> >>>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly. >>>>>>>>>>>>> >>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >>>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >>>>>>>>>>>>> >>>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell. >>>>>>>>>>>>> >>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! >>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce >>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 >>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. >>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 >>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes >>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. >>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. >>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... >>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 >>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. >>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... >>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 >>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing. >>>>>>>>>>>>> >>>>>>>>>>>>> Rebooted >>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>>>>>> Several repeats of the following two lines on the console screen:- >>>>>>>>>>>>> >>>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) >>>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I have given up at this point. >>>>>>>>>>>>> >>>>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know. >>>>>>>>>>>>> >>>>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> >>>>>>>>>>>>> Adolf. >>>
Hi Michael,
On 19/05/2022 13:33, Michael Tremer wrote:
Hello,
That would be brilliant. Let me know how it is going.
Not a good result I am afraid.
You would need to run this first:
https://patchwork.ipfire.org/project/ipfire/patch/20220519085634.197389-1-mi...
I ran that and rd.auto was added to /etc/default/grub
You didn't mention about running grub-mkconfig -o /boot/grub/grub.cfg and I had to run that previously to get rd.auto persistent so I ran that here. Not sure if that was correct or not.
And then just run the script and reboot. The boot process might stall for a minute. Just let it do its job. And the RAID should come up just fine.
I ran the script and cat /proc/mdstat came back with no personalities I then rebooted and after stopping everything the screen went to restart but then came up with the following message:- FATAL: No bootable medium found! System halted
I have a vm clone of the CU167 vm with the failed raid array and with various changes done with only one disk connected so I can clone from this vm and run whatever further changes you identify are needed.
Regards, Adolf.
The script will also avoid the previous issues with the file system that you have outlined after an unsuccessful resync.
-Michael
On 19 May 2022, at 12:21, Adolf Belka adolf.belka@ipfire.org wrote:
Hi,
On 19/05/2022 13:14, Michael Tremer wrote:
Hello,
On 19 May 2022, at 11:57, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 19/05/2022 10:59, Michael Tremer wrote:
Please note the patch that I just posted to fix any broken RAID arrays. Does anyone have any systems that we can use to test this?
Does the script need to be run manually or has it been added to the nightlies together with the rd.auto patch so that I can do a CU168 Testing update evaluation?
My patch includes that it will run automatically with the updater. However, that is not part of the build yet, because I would like it to be confirmed that I do not destroy anything. The whole script is a rather risky operation :)
Okay, clear. Then I will execute the rd.auto change reboot and then run the script.
Will feedback how things go. Adolf.
Regards, Adolf.
-Michael
On 17 May 2022, at 09:09, Adolf Belka adolf.belka@ipfire.org wrote:
Hi Michael,
On 13/05/2022 22:37, Michael Tremer wrote: > Hello, > Okay, this is really bad then if we are breaking RAID installations. > I tried to debug this, but VirtualBox is not my friend today. > I asked some bug reporter to provide me with the logs that I want over here: > https://bugzilla.ipfire.org/show_bug.cgi?id=12862 After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
Thanks very much for all the help.
Regards, Adolf. > -Michael >> On 13 May 2022, at 18:12, Adolf Belka adolf.belka@ipfire.org wrote: >> >> Hi All, >> >> I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go. >> >> I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk. >> >> I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array. >> >> The vm shows the following >> >> -bash-5.1$ cat /proc/mdstat >> Personalities : >> md127 : inactive sdb[1](S) >> 52428724 blocks super 1.0 >> >> unused devices: <none> >> >> >> It looks like IPFire was only running on sda. >> >> So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated. >> >> The CU168 vm that is working has the same as the above setting. >> >> The CU168 vm that is not working has the following setting >> >> -bash-5.1$ cat /proc/mdstat >> Personalities : >> md127 : inactive sda[0](S) >> 52428724 blocks super 1.0 >> >> unused devices: <none> >> >> So it looks like it has booted up on the disk that was not updated from the raid pair. >> >> >> This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode. >> >> It is probably dependent on which of the two disks actually get used to boot from. >> >> So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade. >> >> So raid systems need to have their raid arrays fully working before doing an upgrade. >> >> At least the above is my interpretation of what I have found. What does everyone else think. >> >> >> Regards, >> >> Adolf. >> >> >> On 13/05/2022 15:59, Adolf Belka wrote: >>> Hi Michael, >>> >>> I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules. >>> >>> I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot. >>> >>> However I did copy the boot directory info for these two cases and found an interesting effect. >>> >>> For the problem vm here was the boot directory layout of the CU167 base machine >>> >>> drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . >>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. >>> -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire >>> drwxr-xr-x 3 root root 16K Jan 1 1970 efi >>> drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub >>> -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img >>> drwx------ 2 root root 16K Sep 30 2021 lost+found >>> -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire >>> -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire >>> >>> After upgrade and before reboot this had changed to >>> >>> drwxr-xr-x 5 root root 4.0K May 13 15:40 . >>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. >>> -rw-r--r-- 1 root root 179K May 3 04:02 config-5.15.35-ipfire >>> drwxr-xr-x 3 root root 16K Jan 1 1970 efi >>> drwxr-xr-x 6 root root 4.0K May 13 15:41 grub >>> -rw------- 1 root root 26M May 13 15:41 initramfs-5.15.35-ipfire.img >>> drwx------ 2 root root 16K Sep 30 2021 lost+found >>> -rw-r--r-- 1 root root 4.7M May 3 04:02 System.map-5.15.35-ipfire >>> -rw-r--r-- 1 root root 6.7M May 3 04:02 vmlinuz-5.15.35-ipfire >>> >>> Dates and times of some of the files/directories have changed as would be expected. >>> >>> After Reboot and with the problem the boot directory was now >>> >>> drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . >>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. >>> -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire >>> drwxr-xr-x 3 root root 4.0K Sep 30 2021 efi >>> drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub >>> -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img >>> drwx------ 2 root root 16K Sep 30 2021 lost+found >>> -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire >>> -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire >>> >>> Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed >>> >>> For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade. >>> >>> I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not. >>> >>> Regards, >>> >>> Adolf. >>> >>> On 13/05/2022 12:09, Michael Tremer wrote: >>>> Hello, >>>> >>>>> On 12 May 2022, at 21:10, Adolf Belka adolf.belka@ipfire.org wrote: >>>>> >>>>> Hi Michael, >>>>> >>>>> On 12/05/2022 14:53, Michael Tremer wrote: >>>>>> Hello, >>>>>>> On 12 May 2022, at 12:25, Adolf Belka adolf.belka@ipfire.org wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> On 12/05/2022 11:13, Michael Tremer wrote: >>>>>>>> Hello, >>>>>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while. >>>>>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved. >>>>>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started? >>>>>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that. >>>>>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm. >>>>>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case? >>>>>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces. >>>>>> Okay. That is indeed quite bad then. >>>>>> Since there is no kernel in 168, is there a chance that we broke the update to 167? >>>>> I don't know but as far as I have been concerned CU167 has worked well on my vm. >>>>> Is there something I should look for on my CU167 vm? >>>>>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface. >>>>>> Third time lucky isn’t good enough for me :) >>>>> Me neither :-) >>>>>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces. >>>>>>> >>>>>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers >>>>>> Have there been any modules loaded? If one loads, they all should load. >>>>> This is what I get with lsmod >>>>> >>>>> Module Size Used by >>>>> crct10dif_pclmul 16384 1 >>>>> crc32_pclmul 16384 0 >>>>> ghash_clmulni_intel 16384 0 >>>>> serio_raw 20480 0 >>>>> ohci_pci 20480 0 >>>>> ata_generic 16384 0 >>>>> pata_acpi 16384 0 >>>>> video 57344 0 >>>> >>>> It looks like these modules could have come from the initramdisk image. At least that is working well. >>>> >>>>> On my running CU1678 vm lsmod gives the following >>>>> >>>>> Module Size Used by >>>>> tun 61440 2 >>>>> nfnetlink_queue 28672 1 >>>>> xt_NFQUEUE 16384 8 >>>>> xt_MASQUERADE 20480 1 >>>>> cfg80211 1036288 0 >>>>> rfkill 32768 1 cfg80211 >>>>> 8021q 40960 0 >>>>> garp 16384 1 8021q >>>>> xt_set 16384 260 >>>>> ip_set_hash_net 49152 258 >>>>> ip_set 57344 2 xt_set,ip_set_hash_net >>>>> xt_hashlimit 20480 2 >>>>> xt_multiport 20480 4 >>>>> xt_policy 16384 5 >>>>> xt_TCPMSS 16384 1 >>>>> xt_conntrack 16384 7 >>>>> xt_comment 16384 18 >>>>> ipt_REJECT 16384 1 >>>>> nf_reject_ipv4 16384 1 ipt_REJECT >>>>> xt_LOG 20480 26 >>>>> xt_limit 16384 25 >>>>> xt_mark 16384 27 >>>>> xt_connmark 16384 2 >>>>> nf_log_syslog 24576 26 >>>>> iptable_raw 16384 0 >>>>> iptable_mangle 16384 1 >>>>> iptable_filter 16384 1 >>>>> vfat 24576 1 >>>>> fat 90112 1 vfat >>>>> sch_cake 36864 4 >>>>> intel_powerclamp 20480 0 >>>>> psmouse 184320 0 >>>>> pcspkr 16384 0 >>>>> i2c_piix4 28672 0 >>>>> e1000 163840 0 >>>>> i2c_core 106496 2 psmouse,i2c_piix4 >>>>> crct10dif_pclmul 16384 1 >>>>> crc32_pclmul 16384 0 >>>>> ata_generic 16384 0 >>>>> pata_acpi 16384 0 >>>>> ghash_clmulni_intel 16384 0 >>>>> serio_raw 20480 0 >>>>> ohci_pci 20480 0 >>>>> video 57344 0 >>>> >>>> Yes, this looks more like it. >>>> >>>>> The last 8 entries are the same but a whole load of others are missing. >>>> >>>> Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t. >>>> >>>> I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else. >>>> >>>> -Michael >>>> >>>>> >>>>> Regards, >>>>> Adolf. >>>>>>> Tried modprobe e1000 but this came back with the following error >>>>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service >>>>>>> >>>>>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once. >>>>>>> >>>>>>> Regards >>>>>>> Adolf >>>>>>>>> On 11 May 2022, at 20:08, Adolf Belka adolf.belka@ipfire.org wrote: >>>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> On 11/05/2022 20:48, Jon Murphy wrote: >>>>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info: >>>>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* >>>>>>>>>> Shouldn’t this be Core Update 168? >>>>>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases. >>>>>>>>> >>>>>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm. >>>>>>>>> >>>>>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box. >>>>>>>>> >>>>>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box. >>>>>>>>> >>>>>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box. >>>>>>>> That seems to be a bug that should not be there. Did we recently update apache? >>>>>>>> -Michael >>>>>>>>> >>>>>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem. >>>>>>>>> >>>>>>>>> I think I am going to raise a bug on this. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Adolf. >>>>>>>>>> Jon >>>>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka@ipfire.org mailto:adolf.belka@ipfire.org> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi All, >>>>>>>>>>> >>>>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote: >>>>>>>>>>>> Hi Leo, >>>>>>>>>>>> >>>>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote: >>>>>>>>>>>>> Hi Adolf, >>>>>>>>>>>>> >>>>>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code: >>>>>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that. >>>>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.... https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 >>>>>>>>>>>> Now I see it in the code. Thanks very much. >>>>>>>>>>>>> >>>>>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems! >>>>>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions. >>>>>>>>>>>> >>>>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments. >>>>>>>>>>>> Will wait to hear other inputs on this problem. >>>>>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again. >>>>>>>>>>>> >>>>>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones. >>>>>>>>>>> >>>>>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm. >>>>>>>>>>> >>>>>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/ >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> Adolf. >>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Adolf. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Best regards >>>>>>>>>>>>> Leo >>>>>>>>>>>>> >>>>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> First try:- >>>>>>>>>>>>>> >>>>>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had >>>>>>>>>>>>>> >>>>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c >>>>>>>>>>>>>> >>>>>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message >>>>>>>>>>>>>> >>>>>>>>>>>>>> starting >>>>>>>>>>>>>> DUID 00:04:70:........ >>>>>>>>>>>>>> red0: interface not found >>>>>>>>>>>>>> >>>>>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same >>>>>>>>>>>>>> >>>>>>>>>>>>>> Loading of module with unavailable key is rejected >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Second try:- >>>>>>>>>>>>>> >>>>>>>>>>>>>> This time running the update took over 10 mins. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167. >>>>>>>>>>>>>> >>>>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Repository was set back to Stable. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Before running the pakfire directory had the following >>>>>>>>>>>>>> >>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>>>>>>> >>>>>>>>>>>>>> After running it had >>>>>>>>>>>>>> >>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >>>>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >>>>>>>>>>>>>> >>>>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Rebooted. >>>>>>>>>>>>>> >>>>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c >>>>>>>>>>>>>> >>>>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it >>>>>>>>>>>>>> >>>>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Third try:- >>>>>>>>>>>>>> >>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>>>>>>> >>>>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly. >>>>>>>>>>>>>> >>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >>>>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >>>>>>>>>>>>>> >>>>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell. >>>>>>>>>>>>>> >>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started! >>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce >>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168 >>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine. >>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168 >>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes >>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine. >>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com http://ipfire.earl-net.com/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine. >>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting... >>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0 >>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished. >>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting... >>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0 >>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished. >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies... >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above. >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org http://mirror1.ipfire.org/ (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature... >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine. >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Rebooted >>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>>>>>>> Several repeats of the following two lines on the console screen:- >>>>>>>>>>>>>> >>>>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?) >>>>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I have given up at this point. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know. >>>>>>>>>>>>>> >>>>>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Adolf. >>>>
-
Adolf Belka -
Jon Murphy -
Leo Hofmann -
Michael Tremer -
Peter Müller -
Rob Brewer