From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: Feedback on problems with Core Update 168 Testing Date: Thu, 12 May 2022 13:53:07 +0100 Message-ID: <3C5B2854-D47F-4D92-B943-761BEC763191@ipfire.org> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============3740605589603536427==" List-Id: --===============3740605589603536427== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello, > On 12 May 2022, at 12:25, Adolf Belka wrote: >=20 > Hi, >=20 > On 12/05/2022 11:13, Michael Tremer wrote: >> Hello, >> Thanks for spending so much time on this. We definitely need to improve th= e general update experience since we sometimes seem to break people=E2=80=99s= systems and it is not nice to re-install a firewall from scratch. It will ta= ke a while. >> So what I can say is that the kernel module issues come from when the runn= ing kernel is changed and the kernel is trying to load any modules that now h= ave changed. This fails by design, because we sign our kernel modules. The ke= y is randomly generated at build time and used to sign all modules and it the= n thrown away. For each build, we are using a different, unique key that is n= ot preserved. >> This means that although the kernel modules are of the same version, they = cannot be loaded because the signature check fails. That might also explain w= hy you are seeing so many ipset errors, because the kernel cannot load that m= odule any more. However, we use so much ipset now, why isn=E2=80=99t the modu= le loaded from before the update was started? >> The same goes for any network drivers. I assume you are using virtio or a = generic e1000 network adapter which will have been initialised at boot time. = The kernel should never unload the kernel module for that interface and load = it again later. I have no idea what could have triggered that. > It is a generic e1000 network adapter that is being used by VirtualBox. Con= firmed that on my working CU167 vm. >> No matter what though; after you reboot, the new kernel should be booted b= eing able to load all modules it wants and the system should run absolutely f= ine. Can you confirm that that is at least the case? > No it is not the case. Yesterday when I booted the vm several times the sam= e error messages occurred each time and I always had no network interfaces. Okay. That is indeed quite bad then. Since there is no kernel in 168, is there a chance that we broke the update t= o 167? > Just now I tried booting my third attempt vm again and this time it ran oka= y and I ended up with all my networks assigned and I had network connection. = An IP was assigned by dhcp to the red interface. Third time lucky isn=E2=80=99t good enough for me :) > I then rebooted again and this time the error messages were back. I reboote= d the vm three more times and each time the error messages were there and no = network interfaces. >=20 > I checked with lsmod and the e1000 driver module was not loaded. Confirmed = the e1000 driver directory was not present in /sys/bus/pci/drivers Have there been any modules loaded? If one loads, they all should load. > Tried modprobe e1000 but this came back with the following error > modprobe: ERROR: could not insert 'e1000': Key was rejected by service >=20 > So out of 7 or 8 reboots the vm booted with the network drivers loaded once. >=20 > Regards > Adolf >>> On 11 May 2022, at 20:08, Adolf Belka wrote: >>>=20 >>> Hi All, >>>=20 >>> On 11/05/2022 20:48, Jon Murphy wrote: >>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got = the same build info: >>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d83= 4c* >>>> Shouldn=E2=80=99t this be Core Update 168? >>> That was what I thought but I couldn't precisely remember what it was in = the past with Testing Releases. >>>=20 >>> I then downloaded the Core 168 iso from the nightlies build of master/lat= est/x86_64 and did an install on the First attempt vm. >>>=20 >>> I got to the stage of entering the root and admin passwords. Entered what= I usually do for the vm's and after pressing OK on the admin password screen= I got as message box saying "Problem setting IPFire 'admin' user password". = I pressed the OK button and got another message box saying "Initial setup was= not entirely complete. You must ensure that Setup is properly finished by ru= nning setup again at the shell.". Pressing the OK button on that message box = causes setup to restart but again after entering a valid admin password twice= I got the same "Problem setting IPFire 'admin' user password" message box. >>>=20 >>> I then tried installing CU168 from the same iso onto a clone of my runnin= g CU167 vm. Same result with "Problem setting IPFire 'admin' user password" m= essage box. >>>=20 >>> Then I created a completely new vm from scratch and tried installing CU16= 8 Testing iso and again got "Problem setting IPFire 'admin' user password" me= ssage box. >> That seems to be a bug that should not be there. Did we recently update ap= ache? >> -Michael >>>=20 >>> Don't know if this is related to the problem with the missing interfaces = but if not then it is another problem. >>>=20 >>> I think I am going to raise a bug on this. >>>=20 >>> Regards, >>> Adolf. >>>> Jon >>>>> On May 11, 2022, at 9:19 AM, Adolf Belka > wrote: >>>>>=20 >>>>> Hi All, >>>>>=20 >>>>> On 11/05/2022 16:00, Adolf Belka wrote: >>>>>> Hi Leo, >>>>>>=20 >>>>>> On 11/05/2022 15:26, Leo Hofmann wrote: >>>>>>> Hi Adolf, >>>>>>>=20 >>>>>>> Pakfire always automatically reinstalls the current release before up= dating if you are in the testing branch. See this function in the Pakfire cod= e: >>>>>> So that proves that I haven't been looking so closely on my upgrades i= n the past because I have never noticed that. >>>>>>> https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dblob;f=3Dsrc/pakfire/l= ib/functions.pl;h=3Dd4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=3Drefs/heads/= next#l762 >>>>>> Now I see it in the code. Thanks very much. >>>>>>>=20 >>>>>>> But unfortunately this is all I can contribute, my test system update= d to 168 without problems! >>>>>> Thanks for your help anyway. It's got rid of one of my questions. >>>>>>=20 >>>>>> Just need to understand now why 2 out of 3 updates resulted in a compl= ete loss of the interface assignments. >>>>>> Will wait to hear other inputs on this problem. >>>>>> Maybe will try running setup on the first try I had to confirm that I = can re-assign the interfaces again. >>>>>>=20 >>>>> Running setup does not help. None of the interfaces are available to be= assigned to the red, green etc zones. >>>>>=20 >>>>> I checked the interfaces in the vm setup and they are as previously def= ined and previously working on all earlier CU's of my vm. >>>>>=20 >>>>> When rebooting I did see a fast scrolling message that seemed to say so= mething like Invalid Kernel Argument but that is as much as I was able to see/ >>>>>=20 >>>>> Regards, >>>>>=20 >>>>> Adolf. >>>>>=20 >>>>>> Regards, >>>>>> Adolf. >>>>>>=20 >>>>>>>=20 >>>>>>> Best regards >>>>>>> Leo >>>>>>>=20 >>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>>>>>> Hi All, >>>>>>>>=20 >>>>>>>> I have tried to update my Core Update 167 vm machine three times now= (using a clone) and have had several problems so I thought I would outline w= hat has happened for further thoughts. >>>>>>>>=20 >>>>>>>> Normally my updates go without any real problems. This is the first = time where this is not the case. >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> First try:- >>>>>>>>=20 >>>>>>>> After running the update, which went quite quickly, the bottom of th= e screen had >>>>>>>>=20 >>>>>>>> Core Update 167 Development Build: master/c22d834c >>>>>>>>=20 >>>>>>>> I couldn't remember if this is what is expected or not for Testing r= eleases. >>>>>>>>=20 >>>>>>>> Rebooted the vm and on booting a large number of error messages scro= lled across the console screen. My Virtualbox terminal for IPFire has no scro= ll capability so I can only write on things I saw as they flew past or what w= as on the screen when it stopped. >>>>>>>>=20 >>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>>=20 >>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no acce= ss. >>>>>>>>=20 >>>>>>>> Checked the red0 directory and there were no files present. Tried re= starting red0 and got the following message >>>>>>>>=20 >>>>>>>> starting >>>>>>>> DUID 00:04:70:........ >>>>>>>> red0: interface not found >>>>>>>>=20 >>>>>>>> So I ran ip address show and it came up with only the lo interface. = No red0, green0, blue0 or orange0. >>>>>>>>=20 >>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the= same >>>>>>>>=20 >>>>>>>> Loading of module with unavailable key is rejected >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> Second try:- >>>>>>>>=20 >>>>>>>> This time running the update took over 10 mins. >>>>>>>>=20 >>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 up= grade then it ran the Core Update 168. My IPFire vm was definitely on Core Up= date 167. >>>>>>>>=20 >>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>>>>>>=20 >>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/= log/pakfire/ directory the update-core-upgrade-167 file has been updated as w= ell as there being a 168. Both files had the same last modified date/time. >>>>>>>>=20 >>>>>>>> Rebooted. This time everything came back up okay but the bottom of t= he screen now just said Core Update 167 and there was no update-core-upgrade-= 168 file in the pakfire directory and the 167 version was back to its date of= Apr 28 >>>>>>>>=20 >>>>>>>> Repository was set back to Stable. >>>>>>>>=20 >>>>>>>> Changed again to Testing and it said again that there was an update = from 167 to 168. >>>>>>>>=20 >>>>>>>> Before running the pakfire directory had the following >>>>>>>>=20 >>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>=20 >>>>>>>> After running it had >>>>>>>>=20 >>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >>>>>>>>=20 >>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom= of screen. >>>>>>>>=20 >>>>>>>> Rebooted. >>>>>>>>=20 >>>>>>>> All interfaces present. Pakfire under System Status: says Core-Updat= e-Level: 168 and the bottom of the screen showed Core Update 167 Development = Build: master/c22d834c >>>>>>>>=20 >>>>>>>> /opt/pakfire/db/core/mine has 168 in it >>>>>>>>=20 >>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expec= ted for CU168. >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> Third try:- >>>>>>>>=20 >>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>>>=20 >>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very = quickly. >>>>>>>>=20 >>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >>>>>>>>=20 >>>>>>>> Here is the log file info for pakfire from messages.1.gz and message= s which shows it downloading and upgrading CU167 first then doing CU168 as fa= r as I can tell. >>>>>>>>=20 >>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-= x86_64 started! >>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 second= s old. - DEBUG: noforce >>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 16= 6 to 168 >>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-= 2.27-167.ipfire >>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net= .com (HTTPS) - File: pakfire2/2.27.1-x86_64/pak= s/core-upgrade-2.27-167.ipfire >>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_6= 4/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200= - 200 OK >>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start = checking signature... >>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upg= rade-2.27-167.ipfire is fine. >>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x= 86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upg= rade-168 >>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.= org (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/= meta-core-upgrade-168 >>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_6= 4/meta/meta-core-upgrade-168 has size of 1804 bytes >>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200= - 200 OK >>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start = checking signature... >>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-cor= e-upgrade-168 is fine. >>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x= 86_64/meta/meta-core-upgrade-168 >>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-= 2.27-168.ipfire >>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net= .com (HTTPS) - File: pakfire2/2.27.1-x86_64/pak= s/core-upgrade-2.27-168.ipfire >>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_6= 4/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200= - 200 OK >>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start = checking signature... >>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upg= rade-2.27-168.ipfire is fine. >>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x= 86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decr= ypting... >>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 -= Status: 0 >>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgr= ading files and running post-upgrading scripts... >>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Fini= shed. >>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decr= ypting... >>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 -= Status: 0 >>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgr= ading files and running post-upgrading scripts... >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Fini= shed. >>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 secon= ds old. - DEBUG: noforce >>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving depende= ncies... >>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to instal= l all packages listed above. >>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.= ipfire >>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.= org (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/= wio-1.3.2-15.ipfire >>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_6= 4/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200= - 200 OK >>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start = checking signature... >>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.= 2-15.ipfire is fine. >>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x= 86_64/paks/wio-1.3.2-15.ipfire >>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files a= nd running post-upgrading scripts... >>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. = Closing. >>>>>>>>=20 >>>>>>>> Rebooted >>>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>>> Several repeats of the following two lines on the console screen:- >>>>>>>>=20 >>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' := Table does not exist (do you need to insmod?) >>>>>>>> Perhaps iptables or your kernel needs to be upgraded. >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0,= orange0 not present. >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> I have given up at this point. >>>>>>>>=20 >>>>>>>> I have kept the clones of all three attempts so if you need any addi= tional info from logs or files just let me know. >>>>>>>>=20 >>>>>>>> If this is something obviously wrong that I am doing, please let me = know. If not then I will probably raise a bug on this. >>>>>>>>=20 >>>>>>>> Regards, >>>>>>>>=20 >>>>>>>> Adolf. >>>>>>>>=20 --===============3740605589603536427==--