From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: Feedback on problems with Core Update 168 Testing Date: Thu, 12 May 2022 10:13:40 +0100 Message-ID: <1AF45C97-405B-4065-8A4F-40262DEBB6BA@ipfire.org> In-Reply-To: <9b302f4a-9335-fbd9-9083-ead20a61e872@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============5721260576701089400==" List-Id: --===============5721260576701089400== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello, Thanks for spending so much time on this. We definitely need to improve the g= eneral update experience since we sometimes seem to break people=E2=80=99s sy= stems and it is not nice to re-install a firewall from scratch. It will take = a while. So what I can say is that the kernel module issues come from when the running= kernel is changed and the kernel is trying to load any modules that now have= changed. This fails by design, because we sign our kernel modules. The key i= s randomly generated at build time and used to sign all modules and it then t= hrown away. For each build, we are using a different, unique key that is not = preserved. This means that although the kernel modules are of the same version, they can= not be loaded because the signature check fails. That might also explain why = you are seeing so many ipset errors, because the kernel cannot load that modu= le any more. However, we use so much ipset now, why isn=E2=80=99t the module = loaded from before the update was started? The same goes for any network drivers. I assume you are using virtio or a gen= eric e1000 network adapter which will have been initialised at boot time. The= kernel should never unload the kernel module for that interface and load it = again later. I have no idea what could have triggered that. No matter what though; after you reboot, the new kernel should be booted bein= g able to load all modules it wants and the system should run absolutely fine= . Can you confirm that that is at least the case? > On 11 May 2022, at 20:08, Adolf Belka wrote: >=20 > Hi All, >=20 > On 11/05/2022 20:48, Jon Murphy wrote: >> I just did an update from CU 167 (stable) to CU 168 (testing) and I got th= e same build info: >> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c* >> Shouldn=E2=80=99t this be Core Update 168? > That was what I thought but I couldn't precisely remember what it was in th= e past with Testing Releases. >=20 > I then downloaded the Core 168 iso from the nightlies build of master/lates= t/x86_64 and did an install on the First attempt vm. >=20 > I got to the stage of entering the root and admin passwords. Entered what I= usually do for the vm's and after pressing OK on the admin password screen I= got as message box saying "Problem setting IPFire 'admin' user password". I = pressed the OK button and got another message box saying "Initial setup was n= ot entirely complete. You must ensure that Setup is properly finished by runn= ing setup again at the shell.". Pressing the OK button on that message box ca= uses setup to restart but again after entering a valid admin password twice I= got the same "Problem setting IPFire 'admin' user password" message box. >=20 > I then tried installing CU168 from the same iso onto a clone of my running = CU167 vm. Same result with "Problem setting IPFire 'admin' user password" mes= sage box. >=20 > Then I created a completely new vm from scratch and tried installing CU168 = Testing iso and again got "Problem setting IPFire 'admin' user password" mess= age box. That seems to be a bug that should not be there. Did we recently update apach= e? -Michael >=20 > Don't know if this is related to the problem with the missing interfaces bu= t if not then it is another problem. >=20 > I think I am going to raise a bug on this. >=20 > Regards, > Adolf. >> Jon >>> On May 11, 2022, at 9:19 AM, Adolf Belka > wrote: >>>=20 >>> Hi All, >>>=20 >>> On 11/05/2022 16:00, Adolf Belka wrote: >>>> Hi Leo, >>>>=20 >>>> On 11/05/2022 15:26, Leo Hofmann wrote: >>>>> Hi Adolf, >>>>>=20 >>>>> Pakfire always automatically reinstalls the current release before upda= ting if you are in the testing branch. See this function in the Pakfire code: >>>> So that proves that I haven't been looking so closely on my upgrades in = the past because I have never noticed that. >>>>> https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dblob;f=3Dsrc/pakfire/lib= /functions.pl;h=3Dd4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=3Drefs/heads/ne= xt#l762 >>>> Now I see it in the code. Thanks very much. >>>>>=20 >>>>> But unfortunately this is all I can contribute, my test system updated = to 168 without problems! >>>> Thanks for your help anyway. It's got rid of one of my questions. >>>>=20 >>>> Just need to understand now why 2 out of 3 updates resulted in a complet= e loss of the interface assignments. >>>> Will wait to hear other inputs on this problem. >>>> Maybe will try running setup on the first try I had to confirm that I ca= n re-assign the interfaces again. >>>>=20 >>> Running setup does not help. None of the interfaces are available to be a= ssigned to the red, green etc zones. >>>=20 >>> I checked the interfaces in the vm setup and they are as previously defin= ed and previously working on all earlier CU's of my vm. >>>=20 >>> When rebooting I did see a fast scrolling message that seemed to say some= thing like Invalid Kernel Argument but that is as much as I was able to see/ >>>=20 >>> Regards, >>>=20 >>> Adolf. >>>=20 >>>> Regards, >>>> Adolf. >>>>=20 >>>>>=20 >>>>> Best regards >>>>> Leo >>>>>=20 >>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>>>> Hi All, >>>>>>=20 >>>>>> I have tried to update my Core Update 167 vm machine three times now (= using a clone) and have had several problems so I thought I would outline wha= t has happened for further thoughts. >>>>>>=20 >>>>>> Normally my updates go without any real problems. This is the first ti= me where this is not the case. >>>>>>=20 >>>>>>=20 >>>>>> First try:- >>>>>>=20 >>>>>> After running the update, which went quite quickly, the bottom of the = screen had >>>>>>=20 >>>>>> Core Update 167 Development Build: master/c22d834c >>>>>>=20 >>>>>> I couldn't remember if this is what is expected or not for Testing rel= eases. >>>>>>=20 >>>>>> Rebooted the vm and on booting a large number of error messages scroll= ed across the console screen. My Virtualbox terminal for IPFire has no scroll= capability so I can only write on things I saw as they flew past or what was= on the screen when it stopped. >>>>>>=20 >>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>=20 >>>>>> Tried to access via the WUI and no access. Tried via SSH and no access. >>>>>>=20 >>>>>> Checked the red0 directory and there were no files present. Tried rest= arting red0 and got the following message >>>>>>=20 >>>>>> starting >>>>>> DUID 00:04:70:........ >>>>>> red0: interface not found >>>>>>=20 >>>>>> So I ran ip address show and it came up with only the lo interface. No= red0, green0, blue0 or orange0. >>>>>>=20 >>>>>> Looked in bootlog and this was filled with around 4000 lines all the s= ame >>>>>>=20 >>>>>> Loading of module with unavailable key is rejected >>>>>>=20 >>>>>>=20 >>>>>> Second try:- >>>>>>=20 >>>>>> This time running the update took over 10 mins. >>>>>>=20 >>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgr= ade then it ran the Core Update 168. My IPFire vm was definitely on Core Upda= te 167. >>>>>>=20 >>>>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>>>>=20 >>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/lo= g/pakfire/ directory the update-core-upgrade-167 file has been updated as wel= l as there being a 168. Both files had the same last modified date/time. >>>>>>=20 >>>>>> Rebooted. This time everything came back up okay but the bottom of the= screen now just said Core Update 167 and there was no update-core-upgrade-16= 8 file in the pakfire directory and the 167 version was back to its date of A= pr 28 >>>>>>=20 >>>>>> Repository was set back to Stable. >>>>>>=20 >>>>>> Changed again to Testing and it said again that there was an update fr= om 167 to 168. >>>>>>=20 >>>>>> Before running the pakfire directory had the following >>>>>>=20 >>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>=20 >>>>>> After running it had >>>>>>=20 >>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >>>>>>=20 >>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom o= f screen. >>>>>>=20 >>>>>> Rebooted. >>>>>>=20 >>>>>> All interfaces present. Pakfire under System Status: says Core-Update-= Level: 168 and the bottom of the screen showed Core Update 167 Development Bu= ild: master/c22d834c >>>>>>=20 >>>>>> /opt/pakfire/db/core/mine has 168 in it >>>>>>=20 >>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expecte= d for CU168. >>>>>>=20 >>>>>>=20 >>>>>> Third try:- >>>>>>=20 >>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>=20 >>>>>> Selected Testing and ran update from 167 to 168 which occurred very qu= ickly. >>>>>>=20 >>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >>>>>>=20 >>>>>> Here is the log file info for pakfire from messages.1.gz and messages = which shows it downloading and upgrading CU167 first then doing CU168 as far = as I can tell. >>>>>>=20 >>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x8= 6_64 started! >>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds = old. - DEBUG: noforce >>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 = to 168 >>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.= 27-167.ipfire >>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.c= om (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/= core-upgrade-2.27-167.ipfire >>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/= paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 -= 200 OK >>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start ch= ecking signature... >>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgra= de-2.27-167.ipfire is fine. >>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86= _64/paks/core-upgrade-2.27-167.ipfire >>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgra= de-168 >>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.or= g (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/me= ta-core-upgrade-168 >>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/= meta/meta-core-upgrade-168 has size of 1804 bytes >>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 -= 200 OK >>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start ch= ecking signature... >>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-= upgrade-168 is fine. >>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86= _64/meta/meta-core-upgrade-168 >>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.= 27-168.ipfire >>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.c= om (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/= core-upgrade-2.27-168.ipfire >>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/= paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 -= 200 OK >>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start ch= ecking signature... >>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgra= de-2.27-168.ipfire is fine. >>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86= _64/paks/core-upgrade-2.27-168.ipfire >>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decryp= ting... >>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - S= tatus: 0 >>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrad= ing files and running post-upgrading scripts... >>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finish= ed. >>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decryp= ting... >>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - S= tatus: 0 >>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrad= ing files and running post-upgrading scripts... >>>>>>=20 >>>>>>=20 >>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finish= ed. >>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds= old. - DEBUG: noforce >>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependenc= ies... >>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install = all packages listed above. >>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ip= fire >>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.or= g (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wi= o-1.3.2-15.ipfire >>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/= paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 -= 200 OK >>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start ch= ecking signature... >>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-= 15.ipfire is fine. >>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86= _64/paks/wio-1.3.2-15.ipfire >>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and= running post-upgrading scripts... >>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Cl= osing. >>>>>>=20 >>>>>> Rebooted >>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>> Several repeats of the following two lines on the console screen:- >>>>>>=20 >>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : T= able does not exist (do you need to insmod?) >>>>>> Perhaps iptables or your kernel needs to be upgraded. >>>>>>=20 >>>>>>=20 >>>>>> Only lo interface present. All other interfaces red0, green0, blue0, o= range0 not present. >>>>>>=20 >>>>>>=20 >>>>>> I have given up at this point. >>>>>>=20 >>>>>> I have kept the clones of all three attempts so if you need any additi= onal info from logs or files just let me know. >>>>>>=20 >>>>>> If this is something obviously wrong that I am doing, please let me kn= ow. If not then I will probably raise a bug on this. >>>>>>=20 >>>>>> Regards, >>>>>>=20 >>>>>> Adolf. >>>>>>=20 --===============5721260576701089400==--