From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: Feedback on problems with Core Update 168 Testing Date: Thu, 19 May 2022 12:33:35 +0100 Message-ID: In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============5446044899658308014==" List-Id: --===============5446044899658308014== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello, That would be brilliant. Let me know how it is going. You would need to run this first: https://patchwork.ipfire.org/project/ipfire/patch/20220519085634.197389-1-mic= hael.tremer(a)ipfire.org/ And then just run the script and reboot. The boot process might stall for a m= inute. Just let it do its job. And the RAID should come up just fine. The script will also avoid the previous issues with the file system that you = have outlined after an unsuccessful resync. -Michael > On 19 May 2022, at 12:21, Adolf Belka wrote: >=20 > Hi, >=20 > On 19/05/2022 13:14, Michael Tremer wrote: >> Hello, >>> On 19 May 2022, at 11:57, Adolf Belka wrote: >>>=20 >>> Hi Michael, >>>=20 >>> On 19/05/2022 10:59, Michael Tremer wrote: >>>> Please note the patch that I just posted to fix any broken RAID arrays. >>>> Does anyone have any systems that we can use to test this? >>> Does the script need to be run manually or has it been added to the night= lies together with the rd.auto patch so that I can do a CU168 Testing update = evaluation? >> My patch includes that it will run automatically with the updater. However= , that is not part of the build yet, because I would like it to be confirmed = that I do not destroy anything. The whole script is a rather risky operation = :) > Okay, clear. > Then I will execute the rd.auto change reboot and then run the script. >=20 > Will feedback how things go. > Adolf. >>>=20 >>> Regards, >>> Adolf. >>>=20 >>>> -Michael >>>>> On 17 May 2022, at 09:09, Adolf Belka wrote: >>>>>=20 >>>>> Hi Michael, >>>>>=20 >>>>> On 13/05/2022 22:37, Michael Tremer wrote: >>>>>> Hello, >>>>>> Okay, this is really bad then if we are breaking RAID installations. >>>>>> I tried to debug this, but VirtualBox is not my friend today. >>>>>> I asked some bug reporter to provide me with the logs that I want over= here: >>>>>> https://bugzilla.ipfire.org/show_bug.cgi?id=3D12862 >>>>> After applying this fix to my CU167 vm and proving that it stayed consi= stent I have now done an upgrade to CU168 Testing. This does a CU167 update b= efore doing the CU168 update and I am glad to announce that my previous probl= ems with losing all my network interfaces etc are gone. >>>>>=20 >>>>> My Core Update 168 Development Build: master/bbd4767f successfully rest= arted with a fully active raid array and I will now restart doing my CU168 Te= sting evaluation. >>>>>=20 >>>>> Thanks very much for all the help. >>>>>=20 >>>>> Regards, >>>>> Adolf. >>>>>> -Michael >>>>>>> On 13 May 2022, at 18:12, Adolf Belka wrot= e: >>>>>>>=20 >>>>>>> Hi All, >>>>>>>=20 >>>>>>> I think I may understand what is happening with my vm now. Of course = I could also be wrong :-) but I will give it a go. >>>>>>>=20 >>>>>>> I saw bug#12862 where upgrading from CU166 to CU167 had stopped the r= aid array running so that IPFire was only on one disk. >>>>>>>=20 >>>>>>> I had missed seeing this in my evaluation of CU167 on my vm which doe= s run with a raid array. >>>>>>>=20 >>>>>>> The vm shows the following >>>>>>>=20 >>>>>>> -bash-5.1$ cat /proc/mdstat >>>>>>> Personalities : >>>>>>> md127 : inactive sdb[1](S) >>>>>>> 52428724 blocks super 1.0 >>>>>>>=20 >>>>>>> unused devices: >>>>>>>=20 >>>>>>>=20 >>>>>>> It looks like IPFire was only running on sda. >>>>>>>=20 >>>>>>> So when I am doing an upgrade to CU168 it is likely that only one dis= k of the raid pair is being updated. >>>>>>>=20 >>>>>>> The CU168 vm that is working has the same as the above setting. >>>>>>>=20 >>>>>>> The CU168 vm that is not working has the following setting >>>>>>>=20 >>>>>>> -bash-5.1$ cat /proc/mdstat >>>>>>> Personalities : >>>>>>> md127 : inactive sda[0](S) >>>>>>> 52428724 blocks super 1.0 >>>>>>>=20 >>>>>>> unused devices: >>>>>>>=20 >>>>>>> So it looks like it has booted up on the disk that was not updated fr= om the raid pair. >>>>>>>=20 >>>>>>>=20 >>>>>>> This probably also explains why occasionally I have had a vm that was= not working suddenly start working properly when I reboot it, but booting ag= ain can put it back into non-working mode. >>>>>>>=20 >>>>>>> It is probably dependent on which of the two disks actually get used = to boot from. >>>>>>>=20 >>>>>>> So the problem looks like it would only affect people who have raid s= etups and the problem originated in the CU167 upgrade. >>>>>>>=20 >>>>>>> So raid systems need to have their raid arrays fully working before d= oing an upgrade. >>>>>>>=20 >>>>>>> At least the above is my interpretation of what I have found. What do= es everyone else think. >>>>>>>=20 >>>>>>>=20 >>>>>>> Regards, >>>>>>>=20 >>>>>>> Adolf. >>>>>>>=20 >>>>>>>=20 >>>>>>> On 13/05/2022 15:59, Adolf Belka wrote: >>>>>>>> Hi Michael, >>>>>>>>=20 >>>>>>>> I have found some differences between the vm's that have upgraded wi= th no problem and those that have lost most of the modules. >>>>>>>>=20 >>>>>>>> I did another three vm clones and upgrades and the first two all wen= t with no problems. I was starting to think the issue had gone away but on th= e third one the same problem happened after the reboot. >>>>>>>>=20 >>>>>>>> However I did copy the boot directory info for these two cases and f= ound an interesting effect. >>>>>>>>=20 >>>>>>>> For the problem vm here was the boot directory layout of the CU167 b= ase machine >>>>>>>>=20 >>>>>>>> drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . >>>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. >>>>>>>> -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire >>>>>>>> drwxr-xr-x 3 root root 16K Jan 1 1970 efi >>>>>>>> drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub >>>>>>>> -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.i= mg >>>>>>>> drwx------ 2 root root 16K Sep 30 2021 lost+found >>>>>>>> -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire >>>>>>>> -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire >>>>>>>>=20 >>>>>>>> After upgrade and before reboot this had changed to >>>>>>>>=20 >>>>>>>> drwxr-xr-x 5 root root 4.0K May 13 15:40 . >>>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. >>>>>>>> -rw-r--r-- 1 root root 179K May 3 04:02 config-5.15.35-ipfire >>>>>>>> drwxr-xr-x 3 root root 16K Jan 1 1970 efi >>>>>>>> drwxr-xr-x 6 root root 4.0K May 13 15:41 grub >>>>>>>> -rw------- 1 root root 26M May 13 15:41 initramfs-5.15.35-ipfire.i= mg >>>>>>>> drwx------ 2 root root 16K Sep 30 2021 lost+found >>>>>>>> -rw-r--r-- 1 root root 4.7M May 3 04:02 System.map-5.15.35-ipfire >>>>>>>> -rw-r--r-- 1 root root 6.7M May 3 04:02 vmlinuz-5.15.35-ipfire >>>>>>>>=20 >>>>>>>> Dates and times of some of the files/directories have changed as wou= ld be expected. >>>>>>>>=20 >>>>>>>> After Reboot and with the problem the boot directory was now >>>>>>>>=20 >>>>>>>> drwxr-xr-x 5 root root 4.0K Apr 28 09:47 . >>>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 .. >>>>>>>> -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire >>>>>>>> drwxr-xr-x 3 root root 4.0K Sep 30 2021 efi >>>>>>>> drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub >>>>>>>> -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.i= mg >>>>>>>> drwx------ 2 root root 16K Sep 30 2021 lost+found >>>>>>>> -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire >>>>>>>> -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire >>>>>>>>=20 >>>>>>>> Which is the same as for the original CU167 machine except that the = date/time of the efi directory has changed >>>>>>>>=20 >>>>>>>> For the vm clone that upgraded with no problems the boot directory a= fter reboot was the same as after upgrade. >>>>>>>>=20 >>>>>>>> I don';t understand why or how this has happened but it is certainly= a difference between the vm's having the problem and the ones not. >>>>>>>>=20 >>>>>>>> Regards, >>>>>>>>=20 >>>>>>>> Adolf. >>>>>>>>=20 >>>>>>>> On 13/05/2022 12:09, Michael Tremer wrote: >>>>>>>>> Hello, >>>>>>>>>=20 >>>>>>>>>> On 12 May 2022, at 21:10, Adolf Belka w= rote: >>>>>>>>>>=20 >>>>>>>>>> Hi Michael, >>>>>>>>>>=20 >>>>>>>>>> On 12/05/2022 14:53, Michael Tremer wrote: >>>>>>>>>>> Hello, >>>>>>>>>>>> On 12 May 2022, at 12:25, Adolf Belka = wrote: >>>>>>>>>>>>=20 >>>>>>>>>>>> Hi, >>>>>>>>>>>>=20 >>>>>>>>>>>> On 12/05/2022 11:13, Michael Tremer wrote: >>>>>>>>>>>>> Hello, >>>>>>>>>>>>> Thanks for spending so much time on this. We definitely need to= improve the general update experience since we sometimes seem to break peopl= e=E2=80=99s systems and it is not nice to re-install a firewall from scratch.= It will take a while. >>>>>>>>>>>>> So what I can say is that the kernel module issues come from wh= en the running kernel is changed and the kernel is trying to load any modules= that now have changed. This fails by design, because we sign our kernel modu= les. The key is randomly generated at build time and used to sign all modules= and it then thrown away. For each build, we are using a different, unique ke= y that is not preserved. >>>>>>>>>>>>> This means that although the kernel modules are of the same ver= sion, they cannot be loaded because the signature check fails. That might als= o explain why you are seeing so many ipset errors, because the kernel cannot = load that module any more. However, we use so much ipset now, why isn=E2=80= =99t the module loaded from before the update was started? >>>>>>>>>>>>> The same goes for any network drivers. I assume you are using v= irtio or a generic e1000 network adapter which will have been initialised at = boot time. The kernel should never unload the kernel module for that interfac= e and load it again later. I have no idea what could have triggered that. >>>>>>>>>>>> It is a generic e1000 network adapter that is being used by Virt= ualBox. Confirmed that on my working CU167 vm. >>>>>>>>>>>>> No matter what though; after you reboot, the new kernel should = be booted being able to load all modules it wants and the system should run a= bsolutely fine. Can you confirm that that is at least the case? >>>>>>>>>>>> No it is not the case. Yesterday when I booted the vm several ti= mes the same error messages occurred each time and I always had no network in= terfaces. >>>>>>>>>>> Okay. That is indeed quite bad then. >>>>>>>>>>> Since there is no kernel in 168, is there a chance that we broke = the update to 167? >>>>>>>>>> I don't know but as far as I have been concerned CU167 has worked = well on my vm. >>>>>>>>>> Is there something I should look for on my CU167 vm? >>>>>>>>>>>> Just now I tried booting my third attempt vm again and this time= it ran okay and I ended up with all my networks assigned and I had network c= onnection. An IP was assigned by dhcp to the red interface. >>>>>>>>>>> Third time lucky isn=E2=80=99t good enough for me :) >>>>>>>>>> Me neither :-) >>>>>>>>>>>> I then rebooted again and this time the error messages were back= . I rebooted the vm three more times and each time the error messages were th= ere and no network interfaces. >>>>>>>>>>>>=20 >>>>>>>>>>>> I checked with lsmod and the e1000 driver module was not loaded.= Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers >>>>>>>>>>> Have there been any modules loaded? If one loads, they all should= load. >>>>>>>>>> This is what I get with lsmod >>>>>>>>>>=20 >>>>>>>>>> Module Size Used by >>>>>>>>>> crct10dif_pclmul 16384 1 >>>>>>>>>> crc32_pclmul 16384 0 >>>>>>>>>> ghash_clmulni_intel 16384 0 >>>>>>>>>> serio_raw 20480 0 >>>>>>>>>> ohci_pci 20480 0 >>>>>>>>>> ata_generic 16384 0 >>>>>>>>>> pata_acpi 16384 0 >>>>>>>>>> video 57344 0 >>>>>>>>>=20 >>>>>>>>> It looks like these modules could have come from the initramdisk im= age. At least that is working well. >>>>>>>>>=20 >>>>>>>>>> On my running CU1678 vm lsmod gives the following >>>>>>>>>>=20 >>>>>>>>>> Module Size Used by >>>>>>>>>> tun 61440 2 >>>>>>>>>> nfnetlink_queue 28672 1 >>>>>>>>>> xt_NFQUEUE 16384 8 >>>>>>>>>> xt_MASQUERADE 20480 1 >>>>>>>>>> cfg80211 1036288 0 >>>>>>>>>> rfkill 32768 1 cfg80211 >>>>>>>>>> 8021q 40960 0 >>>>>>>>>> garp 16384 1 8021q >>>>>>>>>> xt_set 16384 260 >>>>>>>>>> ip_set_hash_net 49152 258 >>>>>>>>>> ip_set 57344 2 xt_set,ip_set_hash_net >>>>>>>>>> xt_hashlimit 20480 2 >>>>>>>>>> xt_multiport 20480 4 >>>>>>>>>> xt_policy 16384 5 >>>>>>>>>> xt_TCPMSS 16384 1 >>>>>>>>>> xt_conntrack 16384 7 >>>>>>>>>> xt_comment 16384 18 >>>>>>>>>> ipt_REJECT 16384 1 >>>>>>>>>> nf_reject_ipv4 16384 1 ipt_REJECT >>>>>>>>>> xt_LOG 20480 26 >>>>>>>>>> xt_limit 16384 25 >>>>>>>>>> xt_mark 16384 27 >>>>>>>>>> xt_connmark 16384 2 >>>>>>>>>> nf_log_syslog 24576 26 >>>>>>>>>> iptable_raw 16384 0 >>>>>>>>>> iptable_mangle 16384 1 >>>>>>>>>> iptable_filter 16384 1 >>>>>>>>>> vfat 24576 1 >>>>>>>>>> fat 90112 1 vfat >>>>>>>>>> sch_cake 36864 4 >>>>>>>>>> intel_powerclamp 20480 0 >>>>>>>>>> psmouse 184320 0 >>>>>>>>>> pcspkr 16384 0 >>>>>>>>>> i2c_piix4 28672 0 >>>>>>>>>> e1000 163840 0 >>>>>>>>>> i2c_core 106496 2 psmouse,i2c_piix4 >>>>>>>>>> crct10dif_pclmul 16384 1 >>>>>>>>>> crc32_pclmul 16384 0 >>>>>>>>>> ata_generic 16384 0 >>>>>>>>>> pata_acpi 16384 0 >>>>>>>>>> ghash_clmulni_intel 16384 0 >>>>>>>>>> serio_raw 20480 0 >>>>>>>>>> ohci_pci 20480 0 >>>>>>>>>> video 57344 0 >>>>>>>>>=20 >>>>>>>>> Yes, this looks more like it. >>>>>>>>>=20 >>>>>>>>>> The last 8 entries are the same but a whole load of others are mis= sing. >>>>>>>>>=20 >>>>>>>>> Stefan gave me a call this morning telling me that he tried to repr= oduce this but he couldn=E2=80=99t. >>>>>>>>>=20 >>>>>>>>> I am not sure what we can do here. We urgently need to fix this sin= ce we might break people=E2=80=99s installations. But something seems to work= differently for Adolf than for everybody else. >>>>>>>>>=20 >>>>>>>>> -Michael >>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> Regards, >>>>>>>>>> Adolf. >>>>>>>>>>>> Tried modprobe e1000 but this came back with the following error >>>>>>>>>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by s= ervice >>>>>>>>>>>>=20 >>>>>>>>>>>> So out of 7 or 8 reboots the vm booted with the network drivers = loaded once. >>>>>>>>>>>>=20 >>>>>>>>>>>> Regards >>>>>>>>>>>> Adolf >>>>>>>>>>>>>> On 11 May 2022, at 20:08, Adolf Belka wrote: >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> On 11/05/2022 20:48, Jon Murphy wrote: >>>>>>>>>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing)= and I got the same build info: >>>>>>>>>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: ma= ster/c22d834c* >>>>>>>>>>>>>>> Shouldn=E2=80=99t this be Core Update 168? >>>>>>>>>>>>>> That was what I thought but I couldn't precisely remember what= it was in the past with Testing Releases. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> I then downloaded the Core 168 iso from the nightlies build of= master/latest/x86_64 and did an install on the First attempt vm. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> I got to the stage of entering the root and admin passwords. E= ntered what I usually do for the vm's and after pressing OK on the admin pass= word screen I got as message box saying "Problem setting IPFire 'admin' user = password". I pressed the OK button and got another message box saying "Initia= l setup was not entirely complete. You must ensure that Setup is properly fin= ished by running setup again at the shell.". Pressing the OK button on that m= essage box causes setup to restart but again after entering a valid admin pas= sword twice I got the same "Problem setting IPFire 'admin' user password" mes= sage box. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> I then tried installing CU168 from the same iso onto a clone o= f my running CU167 vm. Same result with "Problem setting IPFire 'admin' user = password" message box. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Then I created a completely new vm from scratch and tried inst= alling CU168 Testing iso and again got "Problem setting IPFire 'admin' user p= assword" message box. >>>>>>>>>>>>> That seems to be a bug that should not be there. Did we recentl= y update apache? >>>>>>>>>>>>> -Michael >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Don't know if this is related to the problem with the missing = interfaces but if not then it is another problem. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> I think I am going to raise a bug on this. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>>> Jon >>>>>>>>>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka > wrote: >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote: >>>>>>>>>>>>>>>>> Hi Leo, >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote: >>>>>>>>>>>>>>>>>> Hi Adolf, >>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>> Pakfire always automatically reinstalls the current releas= e before updating if you are in the testing branch. See this function in the = Pakfire code: >>>>>>>>>>>>>>>>> So that proves that I haven't been looking so closely on my= upgrades in the past because I have never noticed that. >>>>>>>>>>>>>>>>>> https://git.ipfire.org/?p=3Dipfire-2.x.git;a=3Dblob;f=3Dsr= c/pakfire/lib/functions.pl;h=3Dd4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=3D= refs/heads/next#l762 >>>>>>>>>>>>>>>>> Now I see it in the code. Thanks very much. >>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>> But unfortunately this is all I can contribute, my test sy= stem updated to 168 without problems! >>>>>>>>>>>>>>>>> Thanks for your help anyway. It's got rid of one of my ques= tions. >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted= in a complete loss of the interface assignments. >>>>>>>>>>>>>>>>> Will wait to hear other inputs on this problem. >>>>>>>>>>>>>>>>> Maybe will try running setup on the first try I had to conf= irm that I can re-assign the interfaces again. >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> Running setup does not help. None of the interfaces are avai= lable to be assigned to the red, green etc zones. >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> I checked the interfaces in the vm setup and they are as pre= viously defined and previously working on all earlier CU's of my vm. >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> When rebooting I did see a fast scrolling message that seeme= d to say something like Invalid Kernel Argument but that is as much as I was = able to see/ >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>> Adolf. >>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>> Best regards >>>>>>>>>>>>>>>>>> Leo >>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka: >>>>>>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> I have tried to update my Core Update 167 vm machine thre= e times now (using a clone) and have had several problems so I thought I woul= d outline what has happened for further thoughts. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Normally my updates go without any real problems. This is= the first time where this is not the case. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> First try:- >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> After running the update, which went quite quickly, the b= ottom of the screen had >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> I couldn't remember if this is what is expected or not fo= r Testing releases. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Rebooted the vm and on booting a large number of error me= ssages scrolled across the console screen. My Virtualbox terminal for IPFire = has no scroll capability so I can only write on things I saw as they flew pas= t or what was on the screen when it stopped. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get m= ore detail. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH = and no access. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Checked the red0 directory and there were no files presen= t. Tried restarting red0 and got the following message >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> starting >>>>>>>>>>>>>>>>>>> DUID 00:04:70:........ >>>>>>>>>>>>>>>>>>> red0: interface not found >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> So I ran ip address show and it came up with only the lo = interface. No red0, green0, blue0 or orange0. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 li= nes all the same >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Loading of module with unavailable key is rejected >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Second try:- >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> This time running the update took over 10 mins. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Up= date 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely= on Core Update 167. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that i= n the /var/log/pakfire/ directory the update-core-upgrade-167 file has been u= pdated as well as there being a 168. Both files had the same last modified da= te/time. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Rebooted. This time everything came back up okay but the = bottom of the screen now just said Core Update 167 and there was no update-co= re-upgrade-168 file in the pakfire directory and the 167 version was back to = its date of Apr 28 >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Repository was set back to Stable. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Changed again to Testing and it said again that there was= an update from 167 to 168. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Before running the pakfire directory had the following >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgr= ade-166.log >>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgr= ade-167.log >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> After running it had >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgr= ade-166.log >>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgr= ade-167.log >>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgra= de-168.log >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834= c at bottom of screen. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Rebooted. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> All interfaces present. Pakfire under System Status: says= Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 D= evelopment Build: master/c22d834c >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it w= as as expected for CU168. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Third try:- >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgr= ade-167.log >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occ= urred very quickly. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgr= ade-167.log >>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgra= de-168.log >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz = and messages which shows it downloading and upgrading CU167 first then doing = CU168 as far as I can tell. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakf= ire 2.27.1-x86_64 started! >>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db i= s 27 seconds old. - DEBUG: noforce >>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from= release 166 to 168 >>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/co= re-upgrade-2.27-167.ipfire >>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers fo= und in list >>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfi= re.earl-net.com (HTTPS) - File: pakfire2/2.27.1= -x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2= .27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes >>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Statu= s-Code: 200 - 200 OK >>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File recei= ved. Start checking signature... >>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature = of core-upgrade-2.27-167.ipfire is fine. >>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfir= e2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire >>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/me= ta-core-upgrade-168 >>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers fo= und in list >>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirr= or1.ipfire.org (HTTPS) - File: pakfire2/2.27.1-x= 86_64/meta/meta-core-upgrade-168 >>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2= .27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes >>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Statu= s-Code: 200 - 200 OK >>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File recei= ved. Start checking signature... >>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature = of meta-core-upgrade-168 is fine. >>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfir= e2/2.27.1-x86_64/meta/meta-core-upgrade-168 >>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/co= re-upgrade-2.27-168.ipfire >>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers fo= und in list >>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfi= re.earl-net.com (HTTPS) - File: pakfire2/2.27.1= -x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2= .27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes >>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Statu= s-Code: 200 - 200 OK >>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File recei= ved. Start checking signature... >>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature = of core-upgrade-2.27-168.ipfire is fine. >>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfir= e2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire >>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrad= e-167: Decrypting... >>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upg= rade-167 >>>>>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-up= grade-167 - Status: 0 >>>>>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrad= e-167: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrad= e-167: Finished. >>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrad= e-168: Decrypting... >>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upg= rade-168 >>>>>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-up= grade-168 - Status: 0 >>>>>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrad= e-168: Upgrading files and running post-upgrading scripts... >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrad= e-168: Finished. >>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db i= s 127 seconds old. - DEBUG: noforce >>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolv= ing dependencies... >>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are goin= g to install all packages listed above. >>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wi= o-1.3.2-15.ipfire >>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers fo= und in list >>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirr= or1.ipfire.org (HTTPS) - File: pakfire2/2.27.1-x= 86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2= .27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes >>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Statu= s-Code: 200 - 200 OK >>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File recei= ved. Start checking signature... >>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature = of wio-1.3.2-15.ipfire is fine. >>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfir= e2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire >>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decryp= ting... >>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio >>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - S= tatus: 0 >>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrad= ing files and running post-upgrading scripts... >>>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp >>>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finish= ed. >>>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has= finished. Closing. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Rebooted >>>>>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get m= ore detail. >>>>>>>>>>>>>>>>>>> Several repeats of the following two lines on the console= screen:- >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table= `filter' : Table does not exist (do you need to insmod?) >>>>>>>>>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Only lo interface present. All other interfaces red0, gre= en0, blue0, orange0 not present. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> I have given up at this point. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> I have kept the clones of all three attempts so if you ne= ed any additional info from logs or files just let me know. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> If this is something obviously wrong that I am doing, ple= ase let me know. If not then I will probably raise a bug on this. >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>>>>> Adolf. >>>>>>>>>=20 --===============5446044899658308014==--