From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: Feedback on problems with Core Update 168 Testing Date: Thu, 19 May 2022 09:59:07 +0100 Message-ID: <279E5AE9-2439-4B71-91CE-4FD101A9E6AC@ipfire.org> In-Reply-To: <4f61dca9-6acd-d278-8807-42186c507e84@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0080678972430378682==" List-Id: --===============0080678972430378682== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Please note the patch that I just posted to fix any broken RAID arrays. Does anyone have any systems that we can use to test this? -Michael > On 17 May 2022, at 09:09, Adolf Belka wrote: >=20 > Hi Michael, >=20 > On 13/05/2022 22:37, Michael Tremer wrote: >> Hello, >> Okay, this is really bad then if we are breaking RAID installations. >> I tried to debug this, but VirtualBox is not my friend today. >> I asked some bug reporter to provide me with the logs that I want over her= e: >> https://bugzilla.ipfire.org/show_bug.cgi?id=3D12862 > After applying this fix to my CU167 vm and proving that it stayed consisten= t I have now done an upgrade to CU168 Testing. This does a CU167 update befor= e doing the CU168 update and I am glad to announce that my previous problems = with losing all my network interfaces etc are gone. >=20 > My Core Update 168 Development Build: master/bbd4767f successfully restarte= d with a fully active raid array and I will now restart doing my CU168 Testin= g evaluation. >=20 > Thanks very much for all the help. >=20 > Regards, > Adolf. >> -Michael >>> On 13 May 2022, at 18:12, Adolf Belka wrote: >>>=20 >>> Hi All, >>>=20 >>> I think I may understand what is happening with my vm now. Of course I co= uld also be wrong :-) but I will give it a go. >>>=20 >>> I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid = array running so that IPFire was only on one disk. >>>=20 >>> I had missed seeing this in my evaluation of CU167 on my vm which does ru= n with a raid array. >>>=20 >>> The vm shows the following >>>=20 >>> -bash-5.1$ cat /proc/mdstat >>> Personalities : >>> md127 : inactive sdb[1](S) >>> 52428724 blocks super 1.0 >>>=20 >>> unused devices: >>>=20 >>>=20 >>> It looks like IPFire was only running on sda. >>>=20 >>> So when I am doing an upgrade to CU168 it is likely that only one disk of= the raid pair is being updated. >>>=20 >>> The CU168 vm that is working has the same as the above setting. >>>=20 >>> The CU168 vm that is not working has the following setting >>>=20 >>> -bash-5.1$ cat /proc/mdstat >>> Personalities : >>> md127 : inactive sda[0](S) >>> 52428724 blocks super 1.0 >>>=20 >>> unused devices: >>>=20 >>> So it looks like it has booted up on the disk that was not updated from t= he raid pair. >>>=20 >>>=20 >>> This probably also explains why occasionally I have had a vm that was not= working suddenly start working properly when I reboot it, but booting again = can put it back into non-working mode. >>>=20 >>> It is probably dependent on which of the two disks actually get used to b= oot from. >>>=20 >>> So the problem looks like it would only affect people who have raid setup= s and the problem originated in the CU167 upgrade. >>>=20 >>> So raid systems need to have their raid arrays fully working before doing= an upgrade. >>>=20 >>> At least the above is my interpretation of what I have found. What does e= veryone else think. >>>=20 >>>=20 >>> Regards, >>>=20 >>> Adolf. >>>=20 >>>=20 >>> On 13/05/2022 15:59, Adolf Belka wrote: >>>> Hi Michael, >>>>=20 >>>> I have found some differences between the vm's that have upgraded with n= o problem and those that have lost most of the modules. >>>>=20 >>>> I did another three vm clones and upgrades and the first two all went wi= th no problems. I was starting to think the issue had gone away but on the th= ird one the same problem happened after the reboot. >>>>=20 >>>> However I did copy the boot directory info for these two cases and found= an interesting effect. >>>>=20 >>>> For the problem vm here was the boot directory layout of the CU167 base = machine >>>>=20 >>>> drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . >>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. >>>> -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire >>>> drwxr-xr-x 3 root root 16K Jan 1 1970 efi >>>> drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub >>>> -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img >>>> drwx------ 2 root root 16K Sep 30 2021 lost+found >>>> -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire >>>> -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire >>>>=20 >>>> After upgrade and before reboot this had changed to >>>>=20 >>>> drwxr-xr-x 5 root root 4.0K May 13 15:40 . >>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. >>>> -rw-r--r-- 1 root root 179K May 3 04:02 config-5.15.35-ipfire >>>> drwxr-xr-x 3 root root 16K Jan 1 1970 efi >>>> drwxr-xr-x 6 root root 4.0K May 13 15:41 grub >>>> -rw------- 1 root root 26M May 13 15:41 initramfs-5.15.35-ipfire.img >>>> drwx------ 2 root root 16K Sep 30 2021 lost+found >>>> -rw-r--r-- 1 root root 4.7M May 3 04:02 System.map-5.15.35-ipfire >>>> -rw-r--r-- 1 root root 6.7M May 3 04:02 vmlinuz-5.15.35-ipfire >>>>=20 >>>> Dates and times of some of the files/directories have changed as would b= e expected. >>>>=20 >>>> After Reboot and with the problem the boot directory was now >>>>=20 >>>> drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . >>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. >>>> -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire >>>> drwxr-xr-x 3 root root 4.0K Sep 30 2021 efi >>>> drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub >>>> -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img >>>> drwx------ 2 root root 16K Sep 30 2021 lost+found >>>> -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire >>>> -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire >>>>=20 >>>> Which is the same as for the original CU167 machine except that the date= /time of the efi directory has changed >>>>=20 >>>> For the vm clone that upgraded with no problems the boot directory after= reboot was the same as after upgrade. >>>>=20 >>>> I don';t understand why or how this has happened but it is certainly a d= ifference between the vm's having the problem and the ones not. >>>>=20 >>>> Regards, >>>>=20 >>>> Adolf. >>>>=20 >>>> On 13/05/2022 12:09, Michael Tremer wrote: >>>>> Hello, >>>>>=20 >>>>>> On 12 May 2022, at 21:10, Adolf Belka wrote: >>>>>>=20 >>>>>> Hi Michael, >>>>>>=20 >>>>>> On 12/05/2022 14:53, Michael Tremer wrote: >>>>>>> Hello, >>>>>>>> On 12 May 2022, at 12:25, Adolf Belka wro= te: >>>>>>>>=20 >>>>>>>> Hi, >>>>>>>>=20 >>>>>>>> On 12/05/2022 11:13, Michael Tremer wrote: >>>>>>>>> Hello, >>>>>>>>> Thanks for spending so much time on this. We definitely need to imp= rove the general update experience since we sometimes seem to break people=E2= =80=99s systems and it is not nice to re-install a firewall from scratch. It = will take a while. >>>>>>>>> So what I can say is that the kernel module issues come from when t= he running kernel is changed and the kernel is trying to load any modules tha= t now have changed. This fails by design, because we sign our kernel modules.= The key is randomly generated at build time and used to sign all modules and= it then thrown away. For each build, we are using a different, unique key th= at is not preserved. >>>>>>>>> This means that although the kernel modules are of the same version= , they cannot be loaded because the signature check fails. That might also ex= plain why you are seeing so many ipset errors, because the kernel cannot load= that module any more. However, we use so much ipset now, why isn=E2=80=99t t= he module loaded from before the update was started? >>>>>>>>> The same goes for any network drivers. I assume you are using virti= o or a generic e1000 network adapter which will have been initialised at boot= time. The kernel should never unload the kernel module for that interface an= d load it again later. I have no idea what could have triggered that. >>>>>>>> It is a generic e1000 network adapter that is being used by VirtualB= ox. Confirmed that on my working CU167 vm. >>>>>>>>> No matter what though; after you reboot, the new kernel should be b= ooted being able to load all modules it wants and the system should run absol= utely fine. Can you confirm that that is at least the case? >>>>>>>> No it is not the case. Yesterday when I booted the vm several times = the same error messages occurred each time and I always had no network interf= aces. >>>>>>> Okay. That is indeed quite bad then. >>>>>>> Since there is no kernel in 168, is there a chance that we broke the = update to 167? >>>>>> I don't know but as far as I have been concerned CU167 has worked well= on my vm. >>>>>> Is there something I should look for on my CU167 vm? >>>>>>>> Just now I tried booting my third attempt vm again and this time it = ran okay and I ended up with all my networks assigned and I had network conne= ction. An IP was assigned by dhcp to the red interface. >>>>>>> Third time lucky isn=E2=80=99t good enough for me :) >>>>>> Me neither :-) >>>>>>>> I then rebooted again and this time the error messages were back. I = rebooted the vm three more times and each time the error messages were there = and no network interfaces. >>>>>>>>=20 >>>>>>>> I checked with lsmod and the e1000 driver module was not loaded. Con= firmed the e1000 driver directory was not present in /sys/bus/pci/drivers >>>>>>> Have there been any modules loaded? If one loads, they all should loa= d. >>>>>> This is what I get with lsmod >>>>>>=20 >>>>>> Module Size Used by >>>>>> crct10dif_pclmul 16384 1 >>>>>> crc32_pclmul 16384 0 >>>>>> ghash_clmulni_intel 16384 0 >>>>>> serio_raw 20480 0 >>>>>> ohci_pci 20480 0 >>>>>> ata_generic 16384 0 >>>>>> pata_acpi 16384 0 >>>>>> video 57344 0 >>>>>=20 >>>>> It looks like these modules could have come from the initramdisk image.= At least that is working well. >>>>>=20 >>>>>> On my running CU1678 vm lsmod gives the following >>>>>>=20 >>>>>> Module Size Used by >>>>>> tun 61440 2 >>>>>> nfnetlink_queue 28672 1 >>>>>> xt_NFQUEUE 16384 8 >>>>>> xt_MASQUERADE 20480 1 >>>>>> cfg80211 1036288 0 >>>>>> rfkill 32768 1 cfg80211 >>>>>> 8021q 40960 0 >>>>>> garp 16384 1 8021q >>>>>> xt_set 16384 260 >>>>>> ip_set_hash_net 49152 258 >>>>>> ip_set 57344 2 xt_set,ip_set_hash_net >>>>>> xt_hashlimit 20480 2 >>>>>> xt_multiport 20480 4 >>>>>> xt_policy 16384 5 >>>>>> xt_TCPMSS 16384 1 >>>>>> xt_conntrack 16384 7 >>>>>> xt_comment 16384 18 >>>>>> ipt_REJECT 16384 1 >>>>>> nf_reject_ipv4 16384 1 ipt_REJECT >>>>>> xt_LOG 20480 26 >>>>>> xt_limit 16384 25 >>>>>> xt_mark 16384 27 >>>>>> xt_connmark 16384 2 >>>>>> nf_log_syslog 24576 26 >>>>>> iptable_raw 16384 0 >>>>>> iptable_mangle 16384 1 >>>>>> iptable_filter 16384 1 >>>>>> vfat 24576 1 >>>>>> fat 90112 1 vfat >>>>>> sch_cake 36864 4 >>>>>> intel_powerclamp 20480 0 >>>>>> psmouse 184320 0 >>>>>> pcspkr 16384 0 >>>>>> i2c_piix4 28672 0 >>>>>> e1000 163840 0 >>>>>> i2c_core 106496 2 psmouse,i2c_piix4 >>>>>> crct10dif_pclmul 16384 1 >>>>>> crc32_pclmul 16384 0 >>>>>> ata_generic 16384 0 >>>>>> pata_acpi 16384 0 >>>>>> ghash_clmulni_intel 16384 0 >>>>>> serio_raw 20480 0 >>>>>> ohci_pci 20480 0 >>>>>> video 57344 0 >>>>>=20 >>>>> Yes, this looks more like it. >>>>>=20 >>>>>> The last 8 entries are the same but a whole load of others are missing. >>>>>=20 >>>>> Stefan gave me a call this morning telling me that he tried to reproduc= e this but he couldn=E2=80=99t. >>>>>=20 >>>>> I am not sure what we can do here. We urgently need to fix this since w= e might break people=E2=80=99s installations. But something seems to work dif= ferently for Adolf than for everybody else. >>>>>=20 >>>>> -Michael >>>>>=20 >>>>>>=20 >>>>>> Regards, >>>>>> Adolf. >>>>>>>> Tried modprobe e1000 but this came back with the following error >>>>>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by servi= ce >>>>>>>>=20 >>>>>>>> So out of 7 or 8 reboots the vm booted with the network drivers load= ed once. >>>>>>>>=20 >>>>>>>> Regards >>>>>>>> Adolf >>>>>>>>>> On 11 May 2022, at 20:08, Adolf Belka w= rote: >>>>>>>>>>=20 >>>>>>>>>> Hi All, >>>>>>>>>>=20 >>>>>>>>>> On 11/05/2022 20:48, Jon Murphy wrote: >>>>>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and= I got the same build info: >>>>>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master= /c22d834c* >>>>>>>>>>> Shouldn=E2=80=99t this be Core Update 168? >>>>>>>>>> That was what I thought but I couldn't precisely remember what it = was in the past with Testing Releases. >>>>>>>>>>=20 >>>>>>>>>> I then downloaded the Core 168 iso from the nightlies build of mas= ter/latest/x86_64 and did an install on the First attempt vm. >>>>>>>>>>=20 >>>>>>>>>> I got to the stage of entering the root and admin passwords. Enter= ed what I usually do for the vm's and after pressing OK on the admin password= screen I got as message box saying "Problem setting IPFire 'admin' user pass= word". I pressed the OK button and got another message box saying "Initial se= tup was not entirely complete. You must ensure that Setup is properly finishe= d by running setup again at the shell.". Pressing the OK button on that messa= ge box causes setup to restart but again after entering a valid admin passwor= d twice I got the same "Problem setting IPFire 'admin' user password" message= box. >>>>>>>>>>=20 >>>>>>>>>> I then tried installing CU168 from the same iso onto a clone of my= running CU167 vm. Same result with "Problem setting IPFire 'admin' user pass= word" message box. >>>>>>>>>>=20 >>>>>>>>>> Then I created a completely new vm from scratch and tried installi= ng CU168 Testing iso and again got "Problem setting IPFire 'admin' user passw= ord" message box. >>>>>>>>> That seems to be a bug that should not be there. Did we recently up= date apache? >>>>>>>>> -Michael >>>>>>>>>>=20 >>>>>>>>>> Don't know if this is related to the problem with the missing inte= rfaces but if not then it is another problem. >>>>>>>>>>=20 >>>>>>>>>> I think I am going to raise a bug on this. >>>>>>>>>>=20 >>>>>>>>>> Regards, >>>>>>>>>> Adolf. >>>>>>>>>>> Jon >>>>>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka > wrote: >>>>>>>>>>>>=20 >>>>>>>>>>>> Hi All, >>>>>>>>>>>>=20 >>>>>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote: >>>>>>>>>>>>> Hi Leo, >>>>>>>>>>>>>=20 >>>>>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote: >>>>>>>>>>>>>> Hi Adolf, >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Pakfire always automatically reinstalls the current release be= fore updating if you are in the testing branch. See this function in the Pakf= ire code: >>>>>>>>>>>>> So that proves that I haven't been looking so closely on my upg= rades in the past because I have never noticed that. >>>>>>>>>>>>>> https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dblob;f=3Dsrc/pa= kfire/lib/functions.pl;h=3Dd4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=3Drefs= /heads/next#l762 >>>>>>>>>>>>> Now I see it in the code. Thanks very much. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> But unfortunately this is all I can contribute, my test system= updated to 168 without problems! >>>>>>>>>>>>> Thanks for your help anyway. It's got rid of one of my question= s. >>>>>>>>>>>>>=20 >>>>>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in = a complete loss of the interface assignments. >>>>>>>>>>>>> Will wait to hear other inputs on this problem. >>>>>>>>>>>>> Maybe will try running setup on the first try I had to confirm = that I can re-assign the interfaces again. >>>>>>>>>>>>>=20 >>>>>>>>>>>> Running setup does not help. None of the interfaces are availabl= e to be assigned to the red, green etc zones. >>>>>>>>>>>>=20 >>>>>>>>>>>> I checked the interfaces in the vm setup and they are as previou= sly defined and previously working on all earlier CU's of my vm. >>>>>>>>>>>>=20 >>>>>>>>>>>> When rebooting I did see a fast scrolling message that seemed to= say something like Invalid Kernel Argument but that is as much as I was able= to see/ >>>>>>>>>>>>=20 >>>>>>>>>>>> Regards, >>>>>>>>>>>>=20 >>>>>>>>>>>> Adolf. >>>>>>>>>>>>=20 >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>=20 >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Best regards >>>>>>>>>>>>>> Leo >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three ti= mes now (using a clone) and have had several problems so I thought I would ou= tline what has happened for further thoughts. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Normally my updates go without any real problems. This is the= first time where this is not the case. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> First try:- >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> After running the update, which went quite quickly, the botto= m of the screen had >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> I couldn't remember if this is what is expected or not for Te= sting releases. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Rebooted the vm and on booting a large number of error messag= es scrolled across the console screen. My Virtualbox terminal for IPFire has = no scroll capability so I can only write on things I saw as they flew past or= what was on the screen when it stopped. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more = detail. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and = no access. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Checked the red0 directory and there were no files present. T= ried restarting red0 and got the following message >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> starting >>>>>>>>>>>>>>> DUID 00:04:70:........ >>>>>>>>>>>>>>> red0: interface not found >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> So I ran ip address show and it came up with only the lo inte= rface. No red0, green0, blue0 or orange0. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines = all the same >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Loading of module with unavailable key is rejected >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Second try:- >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> This time running the update took over 10 mins. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update= 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on = Core Update 167. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in th= e /var/log/pakfire/ directory the update-core-upgrade-167 file has been updat= ed as well as there being a 168. Both files had the same last modified date/t= ime. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Rebooted. This time everything came back up okay but the bott= om of the screen now just said Core Update 167 and there was no update-core-u= pgrade-168 file in the pakfire directory and the 167 version was back to its = date of Apr 28 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Repository was set back to Stable. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Changed again to Testing and it said again that there was an = update from 167 to 168. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Before running the pakfire directory had the following >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-= 166.log >>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-= 167.log >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> After running it had >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-= 166.log >>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-= 167.log >>>>>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-1= 68.log >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at= bottom of screen. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Rebooted. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Cor= e-Update-Level: 168 and the bottom of the screen showed Core Update 167 Devel= opment Build: master/c22d834c >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was a= s expected for CU168. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Third try:- >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-= 167.log >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurre= d very quickly. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-= 167.log >>>>>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-1= 68.log >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and = messages which shows it downloading and upgrading CU167 first then doing CU16= 8 as far as I can tell. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire = 2.27.1-x86_64 started! >>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27= seconds old. - DEBUG: noforce >>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from rel= ease 166 to 168 >>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-u= pgrade-2.27-167.ipfire >>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found = in list >>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.e= arl-net.com (HTTPS) - File: pakfire2/2.27.1-x86= _64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.= 1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Co= de: 200 - 200 OK >>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received.= Start checking signature... >>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of c= ore-upgrade-2.27-167.ipfire is fine. >>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2= .27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-c= ore-upgrade-168 >>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found = in list >>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.= ipfire.org (HTTPS) - File: pakfire2/2.27.1-x86_6= 4/meta/meta-core-upgrade-168 >>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.= 1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes >>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Co= de: 200 - 200 OK >>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received.= Start checking signature... >>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of m= eta-core-upgrade-168 is fine. >>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2= .27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-u= pgrade-2.27-168.ipfire >>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found = in list >>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.e= arl-net.com (HTTPS) - File: pakfire2/2.27.1-x86= _64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.= 1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Co= de: 200 - 200 OK >>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received.= Start checking signature... >>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of c= ore-upgrade-2.27-168.ipfire is fine. >>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2= .27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-16= 7: Decrypting... >>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade= -167 >>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrad= e-167 - Status: 0 >>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-16= 7: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-16= 7: Finished. >>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-16= 8: Decrypting... >>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade= -168 >>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrad= e-168 - Status: 0 >>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-16= 8: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-16= 8: Finished. >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 12= 7 seconds old. - DEBUG: noforce >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving = dependencies... >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to= install all packages listed above. >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.= 3.2-15.ipfire >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found = in list >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.= ipfire.org (HTTPS) - File: pakfire2/2.27.1-x86_6= 4/paks/wio-1.3.2-15.ipfire >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.= 1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Co= de: 200 - 200 OK >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received.= Start checking signature... >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of w= io-1.3.2-15.ipfire is fine. >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2= .27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting= ... >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Statu= s: 0 >>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading = files and running post-upgrading scripts... >>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished. >>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has fin= ished. Closing. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Rebooted >>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more = detail. >>>>>>>>>>>>>>> Several repeats of the following two lines on the console scr= een:- >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `fi= lter' : Table does not exist (do you need to insmod?) >>>>>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0,= blue0, orange0 not present. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> I have given up at this point. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> I have kept the clones of all three attempts so if you need a= ny additional info from logs or files just let me know. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> If this is something obviously wrong that I am doing, please = let me know. If not then I will probably raise a bug on this. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> Adolf. >>>>>=20 --===============0080678972430378682==--