From mboxrd@z Thu Jan 1 00:00:00 1970 From: Adolf Belka To: development@lists.ipfire.org Subject: Re: Feedback on problems with Core Update 168 Testing Date: Thu, 12 May 2022 13:25:47 +0200 Message-ID: In-Reply-To: <1AF45C97-405B-4065-8A4F-40262DEBB6BA@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2852799043996729613==" List-Id: --===============2852799043996729613== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi, On 12/05/2022 11:13, Michael Tremer wrote: > Hello, >=20 > Thanks for spending so much time on this. We definitely need to improve the= general update experience since we sometimes seem to break people=E2=80=99s = systems and it is not nice to re-install a firewall from scratch. It will tak= e a while. >=20 > So what I can say is that the kernel module issues come from when the runni= ng kernel is changed and the kernel is trying to load any modules that now ha= ve changed. This fails by design, because we sign our kernel modules. The key= is randomly generated at build time and used to sign all modules and it then= thrown away. For each build, we are using a different, unique key that is no= t preserved. >=20 > This means that although the kernel modules are of the same version, they c= annot be loaded because the signature check fails. That might also explain wh= y you are seeing so many ipset errors, because the kernel cannot load that mo= dule any more. However, we use so much ipset now, why isn=E2=80=99t the modul= e loaded from before the update was started? >=20 > The same goes for any network drivers. I assume you are using virtio or a g= eneric e1000 network adapter which will have been initialised at boot time. T= he kernel should never unload the kernel module for that interface and load i= t again later. I have no idea what could have triggered that. It is a generic e1000 network adapter that is being used by VirtualBox. Confi= rmed that on my working CU167 vm. >=20 > No matter what though; after you reboot, the new kernel should be booted be= ing able to load all modules it wants and the system should run absolutely fi= ne. Can you confirm that that is at least the case? No it is not the case. Yesterday when I booted the vm several times the same = error messages occurred each time and I always had no network interfaces. Just now I tried booting my third attempt vm again and this time it ran okay = and I ended up with all my networks assigned and I had network connection. An= IP was assigned by dhcp to the red interface. I then rebooted again and this time the error messages were back. I rebooted = the vm three more times and each time the error messages were there and no ne= twork interfaces. I checked with lsmod and the e1000 driver module was not loaded. Confirmed th= e e1000 driver directory was not present in /sys/bus/pci/drivers Tried modprobe e1000 but this came back with the following error modprobe: ERROR: could not insert 'e1000': Key was rejected by service So out of 7 or 8 reboots the vm booted with the network drivers loaded once. Regards Adolf >=20 >> On 11 May 2022, at 20:08, Adolf Belka wrote: >> >> Hi All, >> >> On 11/05/2022 20:48, Jon Murphy wrote: >>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got t= he same build info: >>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834= c* >>> Shouldn=E2=80=99t this be Core Update 168? >> That was what I thought but I couldn't precisely remember what it was in t= he past with Testing Releases. >> >> I then downloaded the Core 168 iso from the nightlies build of master/late= st/x86_64 and did an install on the First attempt vm. >> >> I got to the stage of entering the root and admin passwords. Entered what = I usually do for the vm's and after pressing OK on the admin password screen = I got as message box saying "Problem setting IPFire 'admin' user password". I= pressed the OK button and got another message box saying "Initial setup was = not entirely complete. You must ensure that Setup is properly finished by run= ning setup again at the shell.". Pressing the OK button on that message box c= auses setup to restart but again after entering a valid admin password twice = I got the same "Problem setting IPFire 'admin' user password" message box. >> >> I then tried installing CU168 from the same iso onto a clone of my running= CU167 vm. Same result with "Problem setting IPFire 'admin' user password" me= ssage box. >> >> Then I created a completely new vm from scratch and tried installing CU168= Testing iso and again got "Problem setting IPFire 'admin' user password" mes= sage box. >=20 > That seems to be a bug that should not be there. Did we recently update apa= che? >=20 > -Michael >=20 >> >> Don't know if this is related to the problem with the missing interfaces b= ut if not then it is another problem. >> >> I think I am going to raise a bug on this. >> >> Regards, >> Adolf. >>> Jon >>>> On May 11, 2022, at 9:19 AM, Adolf Belka > wrote: >>>> >>>> Hi All, >>>> >>>> On 11/05/2022 16:00, Adolf Belka wrote: >>>>> Hi Leo, >>>>> >>>>> On 11/05/2022 15:26, Leo Hofmann wrote: >>>>>> Hi Adolf, >>>>>> >>>>>> Pakfire always automatically reinstalls the current release before upd= ating if you are in the testing branch. See this function in the Pakfire code: >>>>> So that proves that I haven't been looking so closely on my upgrades in= the past because I have never noticed that. >>>>>> https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dblob;f=3Dsrc/pakfire/li= b/functions.pl;h=3Dd4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=3Drefs/heads/n= ext#l762 >>>>> Now I see it in the code. Thanks very much. >>>>>> >>>>>> But unfortunately this is all I can contribute, my test system updated= to 168 without problems! >>>>> Thanks for your help anyway. It's got rid of one of my questions. >>>>> >>>>> Just need to understand now why 2 out of 3 updates resulted in a comple= te loss of the interface assignments. >>>>> Will wait to hear other inputs on this problem. >>>>> Maybe will try running setup on the first try I had to confirm that I c= an re-assign the interfaces again. >>>>> >>>> Running setup does not help. None of the interfaces are available to be = assigned to the red, green etc zones. >>>> >>>> I checked the interfaces in the vm setup and they are as previously defi= ned and previously working on all earlier CU's of my vm. >>>> >>>> When rebooting I did see a fast scrolling message that seemed to say som= ething like Invalid Kernel Argument but that is as much as I was able to see/ >>>> >>>> Regards, >>>> >>>> Adolf. >>>> >>>>> Regards, >>>>> Adolf. >>>>> >>>>>> >>>>>> Best regards >>>>>> Leo >>>>>> >>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>>>>> Hi All, >>>>>>> >>>>>>> I have tried to update my Core Update 167 vm machine three times now = (using a clone) and have had several problems so I thought I would outline wh= at has happened for further thoughts. >>>>>>> >>>>>>> Normally my updates go without any real problems. This is the first t= ime where this is not the case. >>>>>>> >>>>>>> >>>>>>> First try:- >>>>>>> >>>>>>> After running the update, which went quite quickly, the bottom of the= screen had >>>>>>> >>>>>>> Core Update 167 Development Build: master/c22d834c >>>>>>> >>>>>>> I couldn't remember if this is what is expected or not for Testing re= leases. >>>>>>> >>>>>>> Rebooted the vm and on booting a large number of error messages scrol= led across the console screen. My Virtualbox terminal for IPFire has no scrol= l capability so I can only write on things I saw as they flew past or what wa= s on the screen when it stopped. >>>>>>> >>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>> >>>>>>> Tried to access via the WUI and no access. Tried via SSH and no acces= s. >>>>>>> >>>>>>> Checked the red0 directory and there were no files present. Tried res= tarting red0 and got the following message >>>>>>> >>>>>>> starting >>>>>>> DUID 00:04:70:........ >>>>>>> red0: interface not found >>>>>>> >>>>>>> So I ran ip address show and it came up with only the lo interface. N= o red0, green0, blue0 or orange0. >>>>>>> >>>>>>> Looked in bootlog and this was filled with around 4000 lines all the = same >>>>>>> >>>>>>> Loading of module with unavailable key is rejected >>>>>>> >>>>>>> >>>>>>> Second try:- >>>>>>> >>>>>>> This time running the update took over 10 mins. >>>>>>> >>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upg= rade then it ran the Core Update 168. My IPFire vm was definitely on Core Upd= ate 167. >>>>>>> >>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>>>>> >>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/l= og/pakfire/ directory the update-core-upgrade-167 file has been updated as we= ll as there being a 168. Both files had the same last modified date/time. >>>>>>> >>>>>>> Rebooted. This time everything came back up okay but the bottom of th= e screen now just said Core Update 167 and there was no update-core-upgrade-1= 68 file in the pakfire directory and the 167 version was back to its date of = Apr 28 >>>>>>> >>>>>>> Repository was set back to Stable. >>>>>>> >>>>>>> Changed again to Testing and it said again that there was an update f= rom 167 to 168. >>>>>>> >>>>>>> Before running the pakfire directory had the following >>>>>>> >>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>> >>>>>>> After running it had >>>>>>> >>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log >>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log >>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log >>>>>>> >>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom = of screen. >>>>>>> >>>>>>> Rebooted. >>>>>>> >>>>>>> All interfaces present. Pakfire under System Status: says Core-Update= -Level: 168 and the bottom of the screen showed Core Update 167 Development B= uild: master/c22d834c >>>>>>> >>>>>>> /opt/pakfire/db/core/mine has 168 in it >>>>>>> >>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expect= ed for CU168. >>>>>>> >>>>>>> >>>>>>> Third try:- >>>>>>> >>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log >>>>>>> >>>>>>> Selected Testing and ran update from 167 to 168 which occurred very q= uickly. >>>>>>> >>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log >>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log >>>>>>> >>>>>>> Here is the log file info for pakfire from messages.1.gz and messages= which shows it downloading and upgrading CU167 first then doing CU168 as far= as I can tell. >>>>>>> >>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x= 86_64 started! >>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds= old. - DEBUG: noforce >>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166= to 168 >>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2= .27-167.ipfire >>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.= com (HTTPS) - File: pakfire2/2.27.1-x86_64/paks= /core-upgrade-2.27-167.ipfire >>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64= /paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 = - 200 OK >>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start c= hecking signature... >>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgr= ade-2.27-167.ipfire is fine. >>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x8= 6_64/paks/core-upgrade-2.27-167.ipfire >>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgr= ade-168 >>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.o= rg (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/m= eta-core-upgrade-168 >>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64= /meta/meta-core-upgrade-168 has size of 1804 bytes >>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 = - 200 OK >>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start c= hecking signature... >>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core= -upgrade-168 is fine. >>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x8= 6_64/meta/meta-core-upgrade-168 >>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2= .27-168.ipfire >>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.= com (HTTPS) - File: pakfire2/2.27.1-x86_64/paks= /core-upgrade-2.27-168.ipfire >>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64= /paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 = - 200 OK >>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start c= hecking signature... >>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgr= ade-2.27-168.ipfire is fine. >>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x8= 6_64/paks/core-upgrade-2.27-168.ipfire >>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decry= pting... >>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167 >>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - = Status: 0 >>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgra= ding files and running post-upgrading scripts... >>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finis= hed. >>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decry= pting... >>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168 >>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - = Status: 0 >>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgra= ding files and running post-upgrading scripts... >>>>>>> >>>>>>> >>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finis= hed. >>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 second= s old. - DEBUG: noforce >>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependen= cies... >>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install= all packages listed above. >>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.i= pfire >>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list >>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.o= rg (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/w= io-1.3.2-15.ipfire >>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64= /paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 = - 200 OK >>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start c= hecking signature... >>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2= -15.ipfire is fine. >>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x8= 6_64/paks/wio-1.3.2-15.ipfire >>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting... >>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0 >>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files an= d running post-upgrading scripts... >>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. C= losing. >>>>>>> >>>>>>> Rebooted >>>>>>> Lots of ipset restore error but went by too fast to get more detail. >>>>>>> Several repeats of the following two lines on the console screen:- >>>>>>> >>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : = Table does not exist (do you need to insmod?) >>>>>>> Perhaps iptables or your kernel needs to be upgraded. >>>>>>> >>>>>>> >>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, = orange0 not present. >>>>>>> >>>>>>> >>>>>>> I have given up at this point. >>>>>>> >>>>>>> I have kept the clones of all three attempts so if you need any addit= ional info from logs or files just let me know. >>>>>>> >>>>>>> If this is something obviously wrong that I am doing, please let me k= now. If not then I will probably raise a bug on this. >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Adolf. >>>>>>> >=20 --===============2852799043996729613==--