Re: Feedback on problems with Core Update 168 Testing

Re: Feedback on problems with Core Update 168 Testing

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 33649 bytes --]

Hello,

> On 31 May 2022, at 18:36, Peter Müller <peter.mueller(a)ipfire.org> wrote:
> 
> Hello Michael,
> 
>> Hello,
>> 
>>> On 30 May 2022, at 19:57, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>> 
>>> Hello Michael,
>>> 
>>> as promised earlier today:
>>> 
>>>> Hello Adolf,
>>>> 
>>>> thank you for all the extensive testing and reporting back.
>>>> 
>>>> Ah, indeed, the newly introduced Cron job is stil missing due to silly me confusing
>>>> the fcrontab binary and an actual fcrontab. I think the most straight-forward way to
>>>> fix this is to just ship the entire crontab file, and force fcrontab to recreate the
>>>> - yes - fcrontab based on its content.
>>>> 
>>>> @Michael: How does one ship a single file with a Core Update that is not covered by
>>>> any rootfile?
>> 
>> You add it to the “files” list?
>> 
>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=config/rootfiles/core/168/filelists/files;h=159d43d86b0a51d5f1c7583c87f51d34cb147ace;hb=HEAD
>> 
>> I am not sure if I get the question.
> 
> my problem is as follows: With the IDSv4 changes, a new cron job has been introduced,
> and an old one needs to be purged from the crontab. Smilar to what we do in
> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/fcron;h=484deb63d77872798f1b9283f5d461fc92098f46;hb=HEAD#l106,
> I would therefore like to ship the entire config/cron/crontab file, and load it into
> fcron the same way.
> 
> Since config/cron/crontab is not part of any rootfile, I am seeking a way to ship it
> with the Core Update. :-)

It is: https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=config/rootfiles/common/fcron;hb=de5896985ccb3c9c732315ddd17106e5c4b1bafe#l98

It is in /var/spool/cron/root.orig.

That is the file that you will need to ship and then you will have to call “fcrontab -z” so that it will be converted into the binary format that fcron uses.

Just note that any custom cron jobs that users might have created in there will be lost. That is why we normally only edit that file.

> 
> Thanks, and best regards,
> Peter Müller
> 
>> 
>>> 
>>> - ping - :-)
>>> 
>>> Thanks, and best regards,
>>> Peter Müller
>>> 
>>>> 
>>>> Thanks, and best regards,
>>>> Peter Müller
>>>> 
>>>> 
>>>>> Hi All,
>>>>> 
>>>>> Virtually everything I have tested in CU168 looks to be working as expected.
>>>>> 
>>>>> The only problem I have found to-date is the Rules update for Suricata in the Intrusion Prevention System.
>>>>> 
>>>>> The old daily or weekly option is gone and the update should now occur twice per day but that entry is not in the fcrontab in CU168 Testing so no updates are occurring for the rulesets unless you manually force them when the update occurs with no problem.
>>>>> 
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> Adolf
>>>>> 
>>>>> 
>>>>> On 17/05/2022 10:09, Adolf Belka wrote:
>>>>>> Hi Michael,
>>>>>> 
>>>>>> On 13/05/2022 22:37, Michael Tremer wrote:
>>>>>>> Hello,
>>>>>>> 
>>>>>>> Okay, this is really bad then if we are breaking RAID installations.
>>>>>>> 
>>>>>>> I tried to debug this, but VirtualBox is not my friend today.
>>>>>>> 
>>>>>>> I asked some bug reporter to provide me with the logs that I want over here:
>>>>>>> 
>>>>>>> https://bugzilla.ipfire.org/show_bug.cgi?id=12862
>>>>>> After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
>>>>>> 
>>>>>> My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
>>>>>> 
>>>>>> Thanks very much for all the help.
>>>>>> 
>>>>>> Regards,
>>>>>> Adolf.
>>>>>>> 
>>>>>>> -Michael
>>>>>>> 
>>>>>>>> On 13 May 2022, at 18:12, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>> 
>>>>>>>> Hi All,
>>>>>>>> 
>>>>>>>> I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
>>>>>>>> 
>>>>>>>> I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
>>>>>>>> 
>>>>>>>> I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
>>>>>>>> 
>>>>>>>> The vm shows the following
>>>>>>>> 
>>>>>>>> -bash-5.1$ cat /proc/mdstat
>>>>>>>> Personalities :
>>>>>>>> md127 : inactive sdb[1](S)
>>>>>>>> 52428724 blocks super 1.0
>>>>>>>> 
>>>>>>>> unused devices: <none>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> It looks like IPFire was only running on sda.
>>>>>>>> 
>>>>>>>> So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
>>>>>>>> 
>>>>>>>> The CU168 vm that is working has the same as the above setting.
>>>>>>>> 
>>>>>>>> The CU168 vm that is not working has the following setting
>>>>>>>> 
>>>>>>>> -bash-5.1$ cat /proc/mdstat
>>>>>>>> Personalities :
>>>>>>>> md127 : inactive sda[0](S)
>>>>>>>> 52428724 blocks super 1.0
>>>>>>>> 
>>>>>>>> unused devices: <none>
>>>>>>>> 
>>>>>>>> So it looks like it has booted up on the disk that was not updated from the raid pair.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
>>>>>>>> 
>>>>>>>> It is probably dependent on which of the two disks actually get used to boot from.
>>>>>>>> 
>>>>>>>> So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
>>>>>>>> 
>>>>>>>> So raid systems need to have their raid arrays fully working before doing an upgrade.
>>>>>>>> 
>>>>>>>> At least the above is my interpretation of what I have found. What does everyone else think.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> 
>>>>>>>> Adolf.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 13/05/2022 15:59, Adolf Belka wrote:
>>>>>>>>> Hi Michael,
>>>>>>>>> 
>>>>>>>>> I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
>>>>>>>>> 
>>>>>>>>> I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
>>>>>>>>> 
>>>>>>>>> However I did copy the boot directory info for these two cases and found an interesting effect.
>>>>>>>>> 
>>>>>>>>> For the problem vm here was the boot directory layout of the CU167 base machine
>>>>>>>>> 
>>>>>>>>> drwxr-xr-x 5 root root 4.0K Apr 28 09:47 .
>>>>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>>>>>> -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>>>>>>>> drwxr-xr-x 3 root root 16K Jan 1 1970 efi
>>>>>>>>> drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub
>>>>>>>>> -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>>>>>>>> drwx------ 2 root root 16K Sep 30 2021 lost+found
>>>>>>>>> -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>>>>>>>> -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>>>>>>> 
>>>>>>>>> After upgrade and before reboot this had changed to
>>>>>>>>> 
>>>>>>>>> drwxr-xr-x 5 root root 4.0K May 13 15:40 .
>>>>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>>>>>> -rw-r--r-- 1 root root 179K May 3 04:02 config-5.15.35-ipfire
>>>>>>>>> drwxr-xr-x 3 root root 16K Jan 1 1970 efi
>>>>>>>>> drwxr-xr-x 6 root root 4.0K May 13 15:41 grub
>>>>>>>>> -rw------- 1 root root 26M May 13 15:41 initramfs-5.15.35-ipfire.img
>>>>>>>>> drwx------ 2 root root 16K Sep 30 2021 lost+found
>>>>>>>>> -rw-r--r-- 1 root root 4.7M May 3 04:02 System.map-5.15.35-ipfire
>>>>>>>>> -rw-r--r-- 1 root root 6.7M May 3 04:02 vmlinuz-5.15.35-ipfire
>>>>>>>>> 
>>>>>>>>> Dates and times of some of the files/directories have changed as would be expected.
>>>>>>>>> 
>>>>>>>>> After Reboot and with the problem the boot directory was now
>>>>>>>>> 
>>>>>>>>> drwxr-xr-x 5 root root 4.0K Apr 28 09:47 .
>>>>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>>>>>> -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>>>>>>>> drwxr-xr-x 3 root root 4.0K Sep 30 2021 efi
>>>>>>>>> drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub
>>>>>>>>> -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>>>>>>>> drwx------ 2 root root 16K Sep 30 2021 lost+found
>>>>>>>>> -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>>>>>>>> -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>>>>>>> 
>>>>>>>>> Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
>>>>>>>>> 
>>>>>>>>> For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
>>>>>>>>> 
>>>>>>>>> I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> 
>>>>>>>>> Adolf.
>>>>>>>>> 
>>>>>>>>> On 13/05/2022 12:09, Michael Tremer wrote:
>>>>>>>>>> Hello,
>>>>>>>>>> 
>>>>>>>>>>> On 12 May 2022, at 21:10, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Hi Michael,
>>>>>>>>>>> 
>>>>>>>>>>> On 12/05/2022 14:53, Michael Tremer wrote:
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>> On 12 May 2022, at 12:25, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 12/05/2022 11:13, Michael Tremer wrote:
>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
>>>>>>>>>>>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
>>>>>>>>>>>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
>>>>>>>>>>>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
>>>>>>>>>>>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
>>>>>>>>>>>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
>>>>>>>>>>>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
>>>>>>>>>>>> Okay. That is indeed quite bad then.
>>>>>>>>>>>> Since there is no kernel in 168, is there a chance that we broke the update to 167?
>>>>>>>>>>> I don't know but as far as I have been concerned CU167 has worked well on my vm.
>>>>>>>>>>> Is there something I should look for on my CU167 vm?
>>>>>>>>>>>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
>>>>>>>>>>>> Third time lucky isn’t good enough for me :)
>>>>>>>>>>> Me neither :-)
>>>>>>>>>>>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
>>>>>>>>>>>> Have there been any modules loaded? If one loads, they all should load.
>>>>>>>>>>> This is what I get with lsmod
>>>>>>>>>>> 
>>>>>>>>>>> Module Size Used by
>>>>>>>>>>> crct10dif_pclmul 16384 1
>>>>>>>>>>> crc32_pclmul 16384 0
>>>>>>>>>>> ghash_clmulni_intel 16384 0
>>>>>>>>>>> serio_raw 20480 0
>>>>>>>>>>> ohci_pci 20480 0
>>>>>>>>>>> ata_generic 16384 0
>>>>>>>>>>> pata_acpi 16384 0
>>>>>>>>>>> video 57344 0
>>>>>>>>>> 
>>>>>>>>>> It looks like these modules could have come from the initramdisk image. At least that is working well.
>>>>>>>>>> 
>>>>>>>>>>> On my running CU1678 vm lsmod gives the following
>>>>>>>>>>> 
>>>>>>>>>>> Module Size Used by
>>>>>>>>>>> tun 61440 2
>>>>>>>>>>> nfnetlink_queue 28672 1
>>>>>>>>>>> xt_NFQUEUE 16384 8
>>>>>>>>>>> xt_MASQUERADE 20480 1
>>>>>>>>>>> cfg80211 1036288 0
>>>>>>>>>>> rfkill 32768 1 cfg80211
>>>>>>>>>>> 8021q 40960 0
>>>>>>>>>>> garp 16384 1 8021q
>>>>>>>>>>> xt_set 16384 260
>>>>>>>>>>> ip_set_hash_net 49152 258
>>>>>>>>>>> ip_set 57344 2 xt_set,ip_set_hash_net
>>>>>>>>>>> xt_hashlimit 20480 2
>>>>>>>>>>> xt_multiport 20480 4
>>>>>>>>>>> xt_policy 16384 5
>>>>>>>>>>> xt_TCPMSS 16384 1
>>>>>>>>>>> xt_conntrack 16384 7
>>>>>>>>>>> xt_comment 16384 18
>>>>>>>>>>> ipt_REJECT 16384 1
>>>>>>>>>>> nf_reject_ipv4 16384 1 ipt_REJECT
>>>>>>>>>>> xt_LOG 20480 26
>>>>>>>>>>> xt_limit 16384 25
>>>>>>>>>>> xt_mark 16384 27
>>>>>>>>>>> xt_connmark 16384 2
>>>>>>>>>>> nf_log_syslog 24576 26
>>>>>>>>>>> iptable_raw 16384 0
>>>>>>>>>>> iptable_mangle 16384 1
>>>>>>>>>>> iptable_filter 16384 1
>>>>>>>>>>> vfat 24576 1
>>>>>>>>>>> fat 90112 1 vfat
>>>>>>>>>>> sch_cake 36864 4
>>>>>>>>>>> intel_powerclamp 20480 0
>>>>>>>>>>> psmouse 184320 0
>>>>>>>>>>> pcspkr 16384 0
>>>>>>>>>>> i2c_piix4 28672 0
>>>>>>>>>>> e1000 163840 0
>>>>>>>>>>> i2c_core 106496 2 psmouse,i2c_piix4
>>>>>>>>>>> crct10dif_pclmul 16384 1
>>>>>>>>>>> crc32_pclmul 16384 0
>>>>>>>>>>> ata_generic 16384 0
>>>>>>>>>>> pata_acpi 16384 0
>>>>>>>>>>> ghash_clmulni_intel 16384 0
>>>>>>>>>>> serio_raw 20480 0
>>>>>>>>>>> ohci_pci 20480 0
>>>>>>>>>>> video 57344 0
>>>>>>>>>> 
>>>>>>>>>> Yes, this looks more like it.
>>>>>>>>>> 
>>>>>>>>>>> The last 8 entries are the same but a whole load of others are missing.
>>>>>>>>>> 
>>>>>>>>>> Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
>>>>>>>>>> 
>>>>>>>>>> I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
>>>>>>>>>> 
>>>>>>>>>> -Michael
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Regards,
>>>>>>>>>>> Adolf.
>>>>>>>>>>>>> Tried modprobe e1000 but this came back with the following error
>>>>>>>>>>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service
>>>>>>>>>>>>> 
>>>>>>>>>>>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Regards
>>>>>>>>>>>>> Adolf
>>>>>>>>>>>>>>> On 11 May 2022, at 20:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 11/05/2022 20:48, Jon Murphy wrote:
>>>>>>>>>>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
>>>>>>>>>>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c*
>>>>>>>>>>>>>>>> Shouldn’t this be Core Update 168?
>>>>>>>>>>>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>>>> That seems to be a bug that should not be there. Did we recently update apache?
>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I think I am going to raise a bug on this.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka(a)ipfire.org <mailto:adolf.belka(a)ipfire.org>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote:
>>>>>>>>>>>>>>>>>> Hi Leo,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote:
>>>>>>>>>>>>>>>>>>> Hi Adolf,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
>>>>>>>>>>>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
>>>>>>>>>>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 <https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762>
>>>>>>>>>>>>>>>>>> Now I see it in the code. Thanks very much.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
>>>>>>>>>>>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
>>>>>>>>>>>>>>>>>> Will wait to hear other inputs on this problem.
>>>>>>>>>>>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>>>>>> Leo
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>>>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> First try:-
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> starting
>>>>>>>>>>>>>>>>>>>> DUID 00:04:70:........
>>>>>>>>>>>>>>>>>>>> red0: interface not found
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Loading of module with unavailable key is rejected
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Second try:-
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> This time running the update took over 10 mins.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Repository was set back to Stable.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Before running the pakfire directory had the following
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> After running it had
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Rebooted.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Third try:-
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>>>>>>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>>>>>>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>>>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>>>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>>>>>>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>>>>>>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>>>>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Rebooted
>>>>>>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>>>>>>> Several repeats of the following two lines on the console screen:-
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>>>>>>>>>>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I have given up at this point.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Adolf.

Re: Feedback on problems with Core Update 168 Testing

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 34808 bytes --]

Hello Michael,

thanks for your reply.

> Hello,
> 
>> On 31 May 2022, at 18:36, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>
>> Hello Michael,
>>
>>> Hello,
>>>
>>>> On 30 May 2022, at 19:57, Peter Müller <peter.mueller(a)ipfire.org> wrote:
>>>>
>>>> Hello Michael,
>>>>
>>>> as promised earlier today:
>>>>
>>>>> Hello Adolf,
>>>>>
>>>>> thank you for all the extensive testing and reporting back.
>>>>>
>>>>> Ah, indeed, the newly introduced Cron job is stil missing due to silly me confusing
>>>>> the fcrontab binary and an actual fcrontab. I think the most straight-forward way to
>>>>> fix this is to just ship the entire crontab file, and force fcrontab to recreate the
>>>>> - yes - fcrontab based on its content.
>>>>>
>>>>> @Michael: How does one ship a single file with a Core Update that is not covered by
>>>>> any rootfile?
>>>
>>> You add it to the “files” list?
>>>
>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=config/rootfiles/core/168/filelists/files;h=159d43d86b0a51d5f1c7583c87f51d34cb147ace;hb=HEAD
>>>
>>> I am not sure if I get the question.
>>
>> my problem is as follows: With the IDSv4 changes, a new cron job has been introduced,
>> and an old one needs to be purged from the crontab. Smilar to what we do in
>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=lfs/fcron;h=484deb63d77872798f1b9283f5d461fc92098f46;hb=HEAD#l106,
>> I would therefore like to ship the entire config/cron/crontab file, and load it into
>> fcron the same way.
>>
>> Since config/cron/crontab is not part of any rootfile, I am seeking a way to ship it
>> with the Core Update. :-)
> 
> It is: https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=config/rootfiles/common/fcron;hb=de5896985ccb3c9c732315ddd17106e5c4b1bafe#l98
> 
> It is in /var/spool/cron/root.orig.
> 
> That is the file that you will need to ship and then you will have to call “fcrontab -z” so that it will be converted into the binary format that fcron uses.

Ah, I see, thanks for the explanation.

https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=4a4fc8f19a8734a7d92895da3772027550e80f01 now
ships this file and rebuilds fcrontab from scratch while upgrading.

> Just note that any custom cron jobs that users might have created in there will be lost. That is why we normally only edit that file.

I am willing to take that risk. We need IPFire users using RAID to have a manual change at their
systems anyway, and rather than doing some error-prone sed'ing over fcrontab, I think it is better
to reset it to a defined state and add a note regarding this to the changelog for Core Update 168.

The latter is now complete from my point of view. As discussed, I will prepare the release announcement
and pass it on to you for the final steps later today.

Thanks, and best regards,
Peter Müller

> 
>>
>> Thanks, and best regards,
>> Peter Müller
>>
>>>
>>>>
>>>> - ping - :-)
>>>>
>>>> Thanks, and best regards,
>>>> Peter Müller
>>>>
>>>>>
>>>>> Thanks, and best regards,
>>>>> Peter Müller
>>>>>
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> Virtually everything I have tested in CU168 looks to be working as expected.
>>>>>>
>>>>>> The only problem I have found to-date is the Rules update for Suricata in the Intrusion Prevention System.
>>>>>>
>>>>>> The old daily or weekly option is gone and the update should now occur twice per day but that entry is not in the fcrontab in CU168 Testing so no updates are occurring for the rulesets unless you manually force them when the update occurs with no problem.
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Adolf
>>>>>>
>>>>>>
>>>>>> On 17/05/2022 10:09, Adolf Belka wrote:
>>>>>>> Hi Michael,
>>>>>>>
>>>>>>> On 13/05/2022 22:37, Michael Tremer wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> Okay, this is really bad then if we are breaking RAID installations.
>>>>>>>>
>>>>>>>> I tried to debug this, but VirtualBox is not my friend today.
>>>>>>>>
>>>>>>>> I asked some bug reporter to provide me with the logs that I want over here:
>>>>>>>>
>>>>>>>> https://bugzilla.ipfire.org/show_bug.cgi?id=12862
>>>>>>> After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
>>>>>>>
>>>>>>> My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
>>>>>>>
>>>>>>> Thanks very much for all the help.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Adolf.
>>>>>>>>
>>>>>>>> -Michael
>>>>>>>>
>>>>>>>>> On 13 May 2022, at 18:12, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
>>>>>>>>>
>>>>>>>>> I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
>>>>>>>>>
>>>>>>>>> I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
>>>>>>>>>
>>>>>>>>> The vm shows the following
>>>>>>>>>
>>>>>>>>> -bash-5.1$ cat /proc/mdstat
>>>>>>>>> Personalities :
>>>>>>>>> md127 : inactive sdb[1](S)
>>>>>>>>> 52428724 blocks super 1.0
>>>>>>>>>
>>>>>>>>> unused devices: <none>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> It looks like IPFire was only running on sda.
>>>>>>>>>
>>>>>>>>> So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
>>>>>>>>>
>>>>>>>>> The CU168 vm that is working has the same as the above setting.
>>>>>>>>>
>>>>>>>>> The CU168 vm that is not working has the following setting
>>>>>>>>>
>>>>>>>>> -bash-5.1$ cat /proc/mdstat
>>>>>>>>> Personalities :
>>>>>>>>> md127 : inactive sda[0](S)
>>>>>>>>> 52428724 blocks super 1.0
>>>>>>>>>
>>>>>>>>> unused devices: <none>
>>>>>>>>>
>>>>>>>>> So it looks like it has booted up on the disk that was not updated from the raid pair.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
>>>>>>>>>
>>>>>>>>> It is probably dependent on which of the two disks actually get used to boot from.
>>>>>>>>>
>>>>>>>>> So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
>>>>>>>>>
>>>>>>>>> So raid systems need to have their raid arrays fully working before doing an upgrade.
>>>>>>>>>
>>>>>>>>> At least the above is my interpretation of what I have found. What does everyone else think.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Adolf.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 13/05/2022 15:59, Adolf Belka wrote:
>>>>>>>>>> Hi Michael,
>>>>>>>>>>
>>>>>>>>>> I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
>>>>>>>>>>
>>>>>>>>>> I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
>>>>>>>>>>
>>>>>>>>>> However I did copy the boot directory info for these two cases and found an interesting effect.
>>>>>>>>>>
>>>>>>>>>> For the problem vm here was the boot directory layout of the CU167 base machine
>>>>>>>>>>
>>>>>>>>>> drwxr-xr-x 5 root root 4.0K Apr 28 09:47 .
>>>>>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>>>>>>> -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>>>>>>>>> drwxr-xr-x 3 root root 16K Jan 1 1970 efi
>>>>>>>>>> drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub
>>>>>>>>>> -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>>>>>>>>> drwx------ 2 root root 16K Sep 30 2021 lost+found
>>>>>>>>>> -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>>>>>>>>> -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>>>>>>>>
>>>>>>>>>> After upgrade and before reboot this had changed to
>>>>>>>>>>
>>>>>>>>>> drwxr-xr-x 5 root root 4.0K May 13 15:40 .
>>>>>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>>>>>>> -rw-r--r-- 1 root root 179K May 3 04:02 config-5.15.35-ipfire
>>>>>>>>>> drwxr-xr-x 3 root root 16K Jan 1 1970 efi
>>>>>>>>>> drwxr-xr-x 6 root root 4.0K May 13 15:41 grub
>>>>>>>>>> -rw------- 1 root root 26M May 13 15:41 initramfs-5.15.35-ipfire.img
>>>>>>>>>> drwx------ 2 root root 16K Sep 30 2021 lost+found
>>>>>>>>>> -rw-r--r-- 1 root root 4.7M May 3 04:02 System.map-5.15.35-ipfire
>>>>>>>>>> -rw-r--r-- 1 root root 6.7M May 3 04:02 vmlinuz-5.15.35-ipfire
>>>>>>>>>>
>>>>>>>>>> Dates and times of some of the files/directories have changed as would be expected.
>>>>>>>>>>
>>>>>>>>>> After Reboot and with the problem the boot directory was now
>>>>>>>>>>
>>>>>>>>>> drwxr-xr-x 5 root root 4.0K Apr 28 09:47 .
>>>>>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>>>>>>> -rw-r--r-- 1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>>>>>>>>> drwxr-xr-x 3 root root 4.0K Sep 30 2021 efi
>>>>>>>>>> drwxr-xr-x 6 root root 4.0K Apr 28 09:47 grub
>>>>>>>>>> -rw------- 1 root root 26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>>>>>>>>> drwx------ 2 root root 16K Sep 30 2021 lost+found
>>>>>>>>>> -rw-r--r-- 1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>>>>>>>>> -rw-r--r-- 1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>>>>>>>>
>>>>>>>>>> Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
>>>>>>>>>>
>>>>>>>>>> For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
>>>>>>>>>>
>>>>>>>>>> I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Adolf.
>>>>>>>>>>
>>>>>>>>>> On 13/05/2022 12:09, Michael Tremer wrote:
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>>> On 12 May 2022, at 21:10, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>>
>>>>>>>>>>>> On 12/05/2022 14:53, Michael Tremer wrote:
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>> On 12 May 2022, at 12:25, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 12/05/2022 11:13, Michael Tremer wrote:
>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
>>>>>>>>>>>>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
>>>>>>>>>>>>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
>>>>>>>>>>>>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
>>>>>>>>>>>>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
>>>>>>>>>>>>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
>>>>>>>>>>>>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
>>>>>>>>>>>>> Okay. That is indeed quite bad then.
>>>>>>>>>>>>> Since there is no kernel in 168, is there a chance that we broke the update to 167?
>>>>>>>>>>>> I don't know but as far as I have been concerned CU167 has worked well on my vm.
>>>>>>>>>>>> Is there something I should look for on my CU167 vm?
>>>>>>>>>>>>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
>>>>>>>>>>>>> Third time lucky isn’t good enough for me :)
>>>>>>>>>>>> Me neither :-)
>>>>>>>>>>>>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
>>>>>>>>>>>>> Have there been any modules loaded? If one loads, they all should load.
>>>>>>>>>>>> This is what I get with lsmod
>>>>>>>>>>>>
>>>>>>>>>>>> Module Size Used by
>>>>>>>>>>>> crct10dif_pclmul 16384 1
>>>>>>>>>>>> crc32_pclmul 16384 0
>>>>>>>>>>>> ghash_clmulni_intel 16384 0
>>>>>>>>>>>> serio_raw 20480 0
>>>>>>>>>>>> ohci_pci 20480 0
>>>>>>>>>>>> ata_generic 16384 0
>>>>>>>>>>>> pata_acpi 16384 0
>>>>>>>>>>>> video 57344 0
>>>>>>>>>>>
>>>>>>>>>>> It looks like these modules could have come from the initramdisk image. At least that is working well.
>>>>>>>>>>>
>>>>>>>>>>>> On my running CU1678 vm lsmod gives the following
>>>>>>>>>>>>
>>>>>>>>>>>> Module Size Used by
>>>>>>>>>>>> tun 61440 2
>>>>>>>>>>>> nfnetlink_queue 28672 1
>>>>>>>>>>>> xt_NFQUEUE 16384 8
>>>>>>>>>>>> xt_MASQUERADE 20480 1
>>>>>>>>>>>> cfg80211 1036288 0
>>>>>>>>>>>> rfkill 32768 1 cfg80211
>>>>>>>>>>>> 8021q 40960 0
>>>>>>>>>>>> garp 16384 1 8021q
>>>>>>>>>>>> xt_set 16384 260
>>>>>>>>>>>> ip_set_hash_net 49152 258
>>>>>>>>>>>> ip_set 57344 2 xt_set,ip_set_hash_net
>>>>>>>>>>>> xt_hashlimit 20480 2
>>>>>>>>>>>> xt_multiport 20480 4
>>>>>>>>>>>> xt_policy 16384 5
>>>>>>>>>>>> xt_TCPMSS 16384 1
>>>>>>>>>>>> xt_conntrack 16384 7
>>>>>>>>>>>> xt_comment 16384 18
>>>>>>>>>>>> ipt_REJECT 16384 1
>>>>>>>>>>>> nf_reject_ipv4 16384 1 ipt_REJECT
>>>>>>>>>>>> xt_LOG 20480 26
>>>>>>>>>>>> xt_limit 16384 25
>>>>>>>>>>>> xt_mark 16384 27
>>>>>>>>>>>> xt_connmark 16384 2
>>>>>>>>>>>> nf_log_syslog 24576 26
>>>>>>>>>>>> iptable_raw 16384 0
>>>>>>>>>>>> iptable_mangle 16384 1
>>>>>>>>>>>> iptable_filter 16384 1
>>>>>>>>>>>> vfat 24576 1
>>>>>>>>>>>> fat 90112 1 vfat
>>>>>>>>>>>> sch_cake 36864 4
>>>>>>>>>>>> intel_powerclamp 20480 0
>>>>>>>>>>>> psmouse 184320 0
>>>>>>>>>>>> pcspkr 16384 0
>>>>>>>>>>>> i2c_piix4 28672 0
>>>>>>>>>>>> e1000 163840 0
>>>>>>>>>>>> i2c_core 106496 2 psmouse,i2c_piix4
>>>>>>>>>>>> crct10dif_pclmul 16384 1
>>>>>>>>>>>> crc32_pclmul 16384 0
>>>>>>>>>>>> ata_generic 16384 0
>>>>>>>>>>>> pata_acpi 16384 0
>>>>>>>>>>>> ghash_clmulni_intel 16384 0
>>>>>>>>>>>> serio_raw 20480 0
>>>>>>>>>>>> ohci_pci 20480 0
>>>>>>>>>>>> video 57344 0
>>>>>>>>>>>
>>>>>>>>>>> Yes, this looks more like it.
>>>>>>>>>>>
>>>>>>>>>>>> The last 8 entries are the same but a whole load of others are missing.
>>>>>>>>>>>
>>>>>>>>>>> Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
>>>>>>>>>>>
>>>>>>>>>>> I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
>>>>>>>>>>>
>>>>>>>>>>> -Michael
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>> Tried modprobe e1000 but this came back with the following error
>>>>>>>>>>>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>> Adolf
>>>>>>>>>>>>>>>> On 11 May 2022, at 20:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 11/05/2022 20:48, Jon Murphy wrote:
>>>>>>>>>>>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
>>>>>>>>>>>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c*
>>>>>>>>>>>>>>>>> Shouldn’t this be Core Update 168?
>>>>>>>>>>>>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>>>>> That seems to be a bug that should not be there. Did we recently update apache?
>>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I think I am going to raise a bug on this.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka(a)ipfire.org <mailto:adolf.belka(a)ipfire.org>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote:
>>>>>>>>>>>>>>>>>>> Hi Leo,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote:
>>>>>>>>>>>>>>>>>>>> Hi Adolf,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
>>>>>>>>>>>>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
>>>>>>>>>>>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 <https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762>
>>>>>>>>>>>>>>>>>>> Now I see it in the code. Thanks very much.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
>>>>>>>>>>>>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
>>>>>>>>>>>>>>>>>>> Will wait to hear other inputs on this problem.
>>>>>>>>>>>>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>>>>>>> Leo
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>>>>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> First try:-
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> starting
>>>>>>>>>>>>>>>>>>>>> DUID 00:04:70:........
>>>>>>>>>>>>>>>>>>>>> red0: interface not found
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Loading of module with unavailable key is rejected
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Second try:-
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> This time running the update took over 10 mins.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Repository was set back to Stable.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Before running the pakfire directory had the following
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> After running it had
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Rebooted.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Third try:-
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>>>>>>>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>>>>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>>>>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>>>>>>>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>>>>>>>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>>>>>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Rebooted
>>>>>>>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>>>>>>>> Several repeats of the following two lines on the console screen:-
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>>>>>>>>>>>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I have given up at this point.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Adolf.
> 

Re: Feedback on problems with Core Update 168 Testing

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 31914 bytes --]

Hello Michael,

as promised earlier today:

> Hello Adolf,
> 
> thank you for all the extensive testing and reporting back.
> 
> Ah, indeed, the newly introduced Cron job is stil missing due to silly me confusing
> the fcrontab binary and an actual fcrontab. I think the most straight-forward way to
> fix this is to just ship the entire crontab file, and force fcrontab to recreate the
> - yes - fcrontab based on its content.
> 
> @Michael: How does one ship a single file with a Core Update that is not covered by
> any rootfile?

- ping - :-)

Thanks, and best regards,
Peter Müller

> 
> Thanks, and best regards,
> Peter Müller
> 
> 
>> Hi All,
>>
>> Virtually everything I have tested in CU168 looks to be working as expected.
>>
>> The only problem I have found to-date is the Rules update for Suricata in the Intrusion Prevention System.
>>
>> The old daily or weekly option is gone and the update should now occur twice per day but that entry is not in the fcrontab in CU168 Testing so no updates are occurring for the rulesets unless you manually force them when the update occurs with no problem.
>>
>>
>> Regards,
>>
>> Adolf
>>
>>
>> On 17/05/2022 10:09, Adolf Belka wrote:
>>> Hi Michael,
>>>
>>> On 13/05/2022 22:37, Michael Tremer wrote:
>>>> Hello,
>>>>
>>>> Okay, this is really bad then if we are breaking RAID installations.
>>>>
>>>> I tried to debug this, but VirtualBox is not my friend today.
>>>>
>>>> I asked some bug reporter to provide me with the logs that I want over here:
>>>>
>>>>    https://bugzilla.ipfire.org/show_bug.cgi?id=12862
>>> After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
>>>
>>> My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
>>>
>>> Thanks very much for all the help.
>>>
>>> Regards,
>>> Adolf.
>>>>
>>>> -Michael
>>>>
>>>>> On 13 May 2022, at 18:12, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>
>>>>> Hi All,
>>>>>
>>>>> I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
>>>>>
>>>>> I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
>>>>>
>>>>> I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
>>>>>
>>>>> The vm shows the following
>>>>>
>>>>> -bash-5.1$ cat /proc/mdstat
>>>>> Personalities :
>>>>> md127 : inactive sdb[1](S)
>>>>>        52428724 blocks super 1.0
>>>>>
>>>>> unused devices: <none>
>>>>>
>>>>>
>>>>> It looks like IPFire was only running on sda.
>>>>>
>>>>> So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
>>>>>
>>>>> The CU168 vm that is working has the same as the above setting.
>>>>>
>>>>> The CU168 vm that is not working has the following setting
>>>>>
>>>>> -bash-5.1$ cat /proc/mdstat
>>>>> Personalities :
>>>>> md127 : inactive sda[0](S)
>>>>>        52428724 blocks super 1.0
>>>>>
>>>>> unused devices: <none>
>>>>>
>>>>> So it looks like it has booted up on the disk that was not updated from the raid pair.
>>>>>
>>>>>
>>>>> This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
>>>>>
>>>>> It is probably dependent on which of the two disks actually get used to boot from.
>>>>>
>>>>> So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
>>>>>
>>>>> So raid systems need to have their raid arrays fully working before doing an upgrade.
>>>>>
>>>>> At least the above is my interpretation of what I have found. What does everyone else think.
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Adolf.
>>>>>
>>>>>
>>>>> On 13/05/2022 15:59, Adolf Belka wrote:
>>>>>> Hi Michael,
>>>>>>
>>>>>> I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
>>>>>>
>>>>>> I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
>>>>>>
>>>>>> However I did copy the boot directory info for these two cases and found an interesting effect.
>>>>>>
>>>>>> For the problem vm here was the boot directory layout of the CU167 base machine
>>>>>>
>>>>>> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>>> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>>>>> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
>>>>>> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
>>>>>> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>>>> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>>>>> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>>>>
>>>>>> After upgrade and before reboot this had changed to
>>>>>>
>>>>>> drwxr-xr-x  5 root root 4.0K May 13 15:40 .
>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>>> -rw-r--r--  1 root root 179K May  3 04:02 config-5.15.35-ipfire
>>>>>> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
>>>>>> drwxr-xr-x  6 root root 4.0K May 13 15:41 grub
>>>>>> -rw-------  1 root root  26M May 13 15:41 initramfs-5.15.35-ipfire.img
>>>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>>>> -rw-r--r--  1 root root 4.7M May  3 04:02 System.map-5.15.35-ipfire
>>>>>> -rw-r--r--  1 root root 6.7M May  3 04:02 vmlinuz-5.15.35-ipfire
>>>>>>
>>>>>> Dates and times of some of the files/directories have changed as would be expected.
>>>>>>
>>>>>> After Reboot and with the problem the boot directory was now
>>>>>>
>>>>>> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>>> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>>>>> drwxr-xr-x  3 root root 4.0K Sep 30  2021 efi
>>>>>> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
>>>>>> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>>>> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>>>>> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>>>>
>>>>>> Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
>>>>>>
>>>>>> For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
>>>>>>
>>>>>> I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Adolf.
>>>>>>
>>>>>> On 13/05/2022 12:09, Michael Tremer wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>>> On 12 May 2022, at 21:10, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>
>>>>>>>> Hi Michael,
>>>>>>>>
>>>>>>>> On 12/05/2022 14:53, Michael Tremer wrote:
>>>>>>>>> Hello,
>>>>>>>>>> On 12 May 2022, at 12:25, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> On 12/05/2022 11:13, Michael Tremer wrote:
>>>>>>>>>>> Hello,
>>>>>>>>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
>>>>>>>>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
>>>>>>>>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
>>>>>>>>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
>>>>>>>>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
>>>>>>>>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
>>>>>>>>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
>>>>>>>>> Okay. That is indeed quite bad then.
>>>>>>>>> Since there is no kernel in 168, is there a chance that we broke the update to 167?
>>>>>>>> I don't know but as far as I have been concerned CU167 has worked well on my vm.
>>>>>>>> Is there something I should look for on my CU167 vm?
>>>>>>>>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
>>>>>>>>> Third time lucky isn’t good enough for me :)
>>>>>>>> Me neither :-)
>>>>>>>>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
>>>>>>>>>>
>>>>>>>>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
>>>>>>>>> Have there been any modules loaded? If one loads, they all should load.
>>>>>>>> This is what I get with lsmod
>>>>>>>>
>>>>>>>> Module            Size    Used by
>>>>>>>> crct10dif_pclmul    16384    1
>>>>>>>> crc32_pclmul        16384    0
>>>>>>>> ghash_clmulni_intel    16384    0
>>>>>>>> serio_raw        20480    0
>>>>>>>> ohci_pci        20480    0
>>>>>>>> ata_generic        16384    0
>>>>>>>> pata_acpi        16384    0
>>>>>>>> video            57344    0
>>>>>>>
>>>>>>> It looks like these modules could have come from the initramdisk image. At least that is working well.
>>>>>>>
>>>>>>>> On my running CU1678 vm lsmod gives the following
>>>>>>>>
>>>>>>>> Module                  Size  Used by
>>>>>>>> tun                    61440  2
>>>>>>>> nfnetlink_queue        28672  1
>>>>>>>> xt_NFQUEUE             16384  8
>>>>>>>> xt_MASQUERADE          20480  1
>>>>>>>> cfg80211             1036288  0
>>>>>>>> rfkill                 32768  1 cfg80211
>>>>>>>> 8021q                  40960  0
>>>>>>>> garp                   16384  1 8021q
>>>>>>>> xt_set                 16384  260
>>>>>>>> ip_set_hash_net        49152  258
>>>>>>>> ip_set                 57344  2 xt_set,ip_set_hash_net
>>>>>>>> xt_hashlimit           20480  2
>>>>>>>> xt_multiport           20480  4
>>>>>>>> xt_policy              16384  5
>>>>>>>> xt_TCPMSS              16384  1
>>>>>>>> xt_conntrack           16384  7
>>>>>>>> xt_comment             16384  18
>>>>>>>> ipt_REJECT             16384  1
>>>>>>>> nf_reject_ipv4         16384  1 ipt_REJECT
>>>>>>>> xt_LOG                 20480  26
>>>>>>>> xt_limit               16384  25
>>>>>>>> xt_mark                16384  27
>>>>>>>> xt_connmark            16384  2
>>>>>>>> nf_log_syslog          24576  26
>>>>>>>> iptable_raw            16384  0
>>>>>>>> iptable_mangle         16384  1
>>>>>>>> iptable_filter         16384  1
>>>>>>>> vfat                   24576  1
>>>>>>>> fat                    90112  1 vfat
>>>>>>>> sch_cake               36864  4
>>>>>>>> intel_powerclamp       20480  0
>>>>>>>> psmouse               184320  0
>>>>>>>> pcspkr                 16384  0
>>>>>>>> i2c_piix4              28672  0
>>>>>>>> e1000                 163840  0
>>>>>>>> i2c_core              106496  2 psmouse,i2c_piix4
>>>>>>>> crct10dif_pclmul       16384  1
>>>>>>>> crc32_pclmul           16384  0
>>>>>>>> ata_generic            16384  0
>>>>>>>> pata_acpi              16384  0
>>>>>>>> ghash_clmulni_intel    16384  0
>>>>>>>> serio_raw              20480  0
>>>>>>>> ohci_pci               20480  0
>>>>>>>> video                  57344  0
>>>>>>>
>>>>>>> Yes, this looks more like it.
>>>>>>>
>>>>>>>> The last 8 entries are the same but a whole load of others are missing.
>>>>>>>
>>>>>>> Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
>>>>>>>
>>>>>>> I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
>>>>>>>
>>>>>>> -Michael
>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Adolf.
>>>>>>>>>> Tried modprobe e1000 but this came back with the following error
>>>>>>>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service
>>>>>>>>>>
>>>>>>>>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> Adolf
>>>>>>>>>>>> On 11 May 2022, at 20:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>
>>>>>>>>>>>> On 11/05/2022 20:48, Jon Murphy wrote:
>>>>>>>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
>>>>>>>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c*
>>>>>>>>>>>>> Shouldn’t this be Core Update 168?
>>>>>>>>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
>>>>>>>>>>>>
>>>>>>>>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
>>>>>>>>>>>>
>>>>>>>>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>>
>>>>>>>>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>>
>>>>>>>>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>> That seems to be a bug that should not be there. Did we recently update apache?
>>>>>>>>>>> -Michael
>>>>>>>>>>>>
>>>>>>>>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
>>>>>>>>>>>>
>>>>>>>>>>>> I think I am going to raise a bug on this.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka(a)ipfire.org <mailto:adolf.belka(a)ipfire.org>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote:
>>>>>>>>>>>>>>> Hi Leo,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote:
>>>>>>>>>>>>>>>> Hi Adolf,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
>>>>>>>>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
>>>>>>>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 <https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762>
>>>>>>>>>>>>>>> Now I see it in the code. Thanks very much.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
>>>>>>>>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
>>>>>>>>>>>>>>> Will wait to hear other inputs on this problem.
>>>>>>>>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>>> Leo
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> First try:-
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> starting
>>>>>>>>>>>>>>>>> DUID 00:04:70:........
>>>>>>>>>>>>>>>>> red0: interface not found
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Loading of module with unavailable key is rejected
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Second try:-
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This time running the update took over 10 mins.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Repository was set back to Stable.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Before running the pakfire directory had the following
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> After running it had
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Rebooted.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Third try:-
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>>>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>>>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>>>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>>>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Rebooted
>>>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>>>> Several repeats of the following two lines on the console screen:-
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>>>>>>>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I have given up at this point.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>
>>>>

Re: Feedback on problems with Core Update 168 Testing

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 33790 bytes --]

Hi Michael,

On 19/05/2022 13:33, Michael Tremer wrote:
> Hello,
> 
> That would be brilliant. Let me know how it is going.
Not a good result I am afraid.
> 
> You would need to run this first:
> 
> https://patchwork.ipfire.org/project/ipfire/patch/20220519085634.197389-1-michael.tremer(a)ipfire.org/
I ran that and rd.auto was added to /etc/default/grub

You didn't mention about running grub-mkconfig -o /boot/grub/grub.cfg and I had to run that previously to get rd.auto persistent so I ran that here. Not sure if that was correct or not.
> 
> And then just run the script and reboot. The boot process might stall for a minute. Just let it do its job. And the RAID should come up just fine.
I ran the script and cat /proc/mdstat came back with no personalities
I then rebooted and after stopping everything the screen went to restart but then came up with the following message:-
FATAL: No bootable medium found! System halted

I have a vm clone of the CU167 vm with the failed raid array and with various changes done with only one disk connected so I can clone from this vm and run whatever further changes you identify are needed.

Regards,
Adolf.
> 
> The script will also avoid the previous issues with the file system that you have outlined after an unsuccessful resync.
> 
> -Michael
> 
>> On 19 May 2022, at 12:21, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi,
>>
>> On 19/05/2022 13:14, Michael Tremer wrote:
>>> Hello,
>>>> On 19 May 2022, at 11:57, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>
>>>> Hi Michael,
>>>>
>>>> On 19/05/2022 10:59, Michael Tremer wrote:
>>>>> Please note the patch that I just posted to fix any broken RAID arrays.
>>>>> Does anyone have any systems that we can use to test this?
>>>> Does the script need to be run manually or has it been added to the nightlies together with the rd.auto patch so that I can do a CU168 Testing update evaluation?
>>> My patch includes that it will run automatically with the updater. However, that is not part of the build yet, because I would like it to be confirmed that I do not destroy anything. The whole script is a rather risky operation :)
>> Okay, clear.
>> Then I will execute the rd.auto change reboot and then run the script.
>>
>> Will feedback how things go.
>> Adolf.
>>>>
>>>> Regards,
>>>> Adolf.
>>>>
>>>>> -Michael
>>>>>> On 17 May 2022, at 09:09, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>
>>>>>> Hi Michael,
>>>>>>
>>>>>> On 13/05/2022 22:37, Michael Tremer wrote:
>>>>>>> Hello,
>>>>>>> Okay, this is really bad then if we are breaking RAID installations.
>>>>>>> I tried to debug this, but VirtualBox is not my friend today.
>>>>>>> I asked some bug reporter to provide me with the logs that I want over here:
>>>>>>>    https://bugzilla.ipfire.org/show_bug.cgi?id=12862
>>>>>> After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
>>>>>>
>>>>>> My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
>>>>>>
>>>>>> Thanks very much for all the help.
>>>>>>
>>>>>> Regards,
>>>>>> Adolf.
>>>>>>> -Michael
>>>>>>>> On 13 May 2022, at 18:12, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
>>>>>>>>
>>>>>>>> I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
>>>>>>>>
>>>>>>>> I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
>>>>>>>>
>>>>>>>> The vm shows the following
>>>>>>>>
>>>>>>>> -bash-5.1$ cat /proc/mdstat
>>>>>>>> Personalities :
>>>>>>>> md127 : inactive sdb[1](S)
>>>>>>>>        52428724 blocks super 1.0
>>>>>>>>
>>>>>>>> unused devices: <none>
>>>>>>>>
>>>>>>>>
>>>>>>>> It looks like IPFire was only running on sda.
>>>>>>>>
>>>>>>>> So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
>>>>>>>>
>>>>>>>> The CU168 vm that is working has the same as the above setting.
>>>>>>>>
>>>>>>>> The CU168 vm that is not working has the following setting
>>>>>>>>
>>>>>>>> -bash-5.1$ cat /proc/mdstat
>>>>>>>> Personalities :
>>>>>>>> md127 : inactive sda[0](S)
>>>>>>>>        52428724 blocks super 1.0
>>>>>>>>
>>>>>>>> unused devices: <none>
>>>>>>>>
>>>>>>>> So it looks like it has booted up on the disk that was not updated from the raid pair.
>>>>>>>>
>>>>>>>>
>>>>>>>> This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
>>>>>>>>
>>>>>>>> It is probably dependent on which of the two disks actually get used to boot from.
>>>>>>>>
>>>>>>>> So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
>>>>>>>>
>>>>>>>> So raid systems need to have their raid arrays fully working before doing an upgrade.
>>>>>>>>
>>>>>>>> At least the above is my interpretation of what I have found. What does everyone else think.
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Adolf.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 13/05/2022 15:59, Adolf Belka wrote:
>>>>>>>>> Hi Michael,
>>>>>>>>>
>>>>>>>>> I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
>>>>>>>>>
>>>>>>>>> I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
>>>>>>>>>
>>>>>>>>> However I did copy the boot directory info for these two cases and found an interesting effect.
>>>>>>>>>
>>>>>>>>> For the problem vm here was the boot directory layout of the CU167 base machine
>>>>>>>>>
>>>>>>>>> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
>>>>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>>>>>> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>>>>>>>> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
>>>>>>>>> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
>>>>>>>>> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>>>>>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>>>>>>> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>>>>>>>> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>>>>>>>
>>>>>>>>> After upgrade and before reboot this had changed to
>>>>>>>>>
>>>>>>>>> drwxr-xr-x  5 root root 4.0K May 13 15:40 .
>>>>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>>>>>> -rw-r--r--  1 root root 179K May  3 04:02 config-5.15.35-ipfire
>>>>>>>>> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
>>>>>>>>> drwxr-xr-x  6 root root 4.0K May 13 15:41 grub
>>>>>>>>> -rw-------  1 root root  26M May 13 15:41 initramfs-5.15.35-ipfire.img
>>>>>>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>>>>>>> -rw-r--r--  1 root root 4.7M May  3 04:02 System.map-5.15.35-ipfire
>>>>>>>>> -rw-r--r--  1 root root 6.7M May  3 04:02 vmlinuz-5.15.35-ipfire
>>>>>>>>>
>>>>>>>>> Dates and times of some of the files/directories have changed as would be expected.
>>>>>>>>>
>>>>>>>>> After Reboot and with the problem the boot directory was now
>>>>>>>>>
>>>>>>>>> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
>>>>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>>>>>> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>>>>>>>> drwxr-xr-x  3 root root 4.0K Sep 30  2021 efi
>>>>>>>>> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
>>>>>>>>> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>>>>>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>>>>>>> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>>>>>>>> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>>>>>>>
>>>>>>>>> Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
>>>>>>>>>
>>>>>>>>> For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
>>>>>>>>>
>>>>>>>>> I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Adolf.
>>>>>>>>>
>>>>>>>>> On 13/05/2022 12:09, Michael Tremer wrote:
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>>> On 12 May 2022, at 21:10, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Michael,
>>>>>>>>>>>
>>>>>>>>>>> On 12/05/2022 14:53, Michael Tremer wrote:
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>> On 12 May 2022, at 12:25, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 12/05/2022 11:13, Michael Tremer wrote:
>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
>>>>>>>>>>>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
>>>>>>>>>>>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
>>>>>>>>>>>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
>>>>>>>>>>>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
>>>>>>>>>>>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
>>>>>>>>>>>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
>>>>>>>>>>>> Okay. That is indeed quite bad then.
>>>>>>>>>>>> Since there is no kernel in 168, is there a chance that we broke the update to 167?
>>>>>>>>>>> I don't know but as far as I have been concerned CU167 has worked well on my vm.
>>>>>>>>>>> Is there something I should look for on my CU167 vm?
>>>>>>>>>>>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
>>>>>>>>>>>> Third time lucky isn’t good enough for me :)
>>>>>>>>>>> Me neither :-)
>>>>>>>>>>>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
>>>>>>>>>>>> Have there been any modules loaded? If one loads, they all should load.
>>>>>>>>>>> This is what I get with lsmod
>>>>>>>>>>>
>>>>>>>>>>> Module            Size    Used by
>>>>>>>>>>> crct10dif_pclmul    16384    1
>>>>>>>>>>> crc32_pclmul        16384    0
>>>>>>>>>>> ghash_clmulni_intel    16384    0
>>>>>>>>>>> serio_raw        20480    0
>>>>>>>>>>> ohci_pci        20480    0
>>>>>>>>>>> ata_generic        16384    0
>>>>>>>>>>> pata_acpi        16384    0
>>>>>>>>>>> video            57344    0
>>>>>>>>>>
>>>>>>>>>> It looks like these modules could have come from the initramdisk image. At least that is working well.
>>>>>>>>>>
>>>>>>>>>>> On my running CU1678 vm lsmod gives the following
>>>>>>>>>>>
>>>>>>>>>>> Module                  Size  Used by
>>>>>>>>>>> tun                    61440  2
>>>>>>>>>>> nfnetlink_queue        28672  1
>>>>>>>>>>> xt_NFQUEUE             16384  8
>>>>>>>>>>> xt_MASQUERADE          20480  1
>>>>>>>>>>> cfg80211             1036288  0
>>>>>>>>>>> rfkill                 32768  1 cfg80211
>>>>>>>>>>> 8021q                  40960  0
>>>>>>>>>>> garp                   16384  1 8021q
>>>>>>>>>>> xt_set                 16384  260
>>>>>>>>>>> ip_set_hash_net        49152  258
>>>>>>>>>>> ip_set                 57344  2 xt_set,ip_set_hash_net
>>>>>>>>>>> xt_hashlimit           20480  2
>>>>>>>>>>> xt_multiport           20480  4
>>>>>>>>>>> xt_policy              16384  5
>>>>>>>>>>> xt_TCPMSS              16384  1
>>>>>>>>>>> xt_conntrack           16384  7
>>>>>>>>>>> xt_comment             16384  18
>>>>>>>>>>> ipt_REJECT             16384  1
>>>>>>>>>>> nf_reject_ipv4         16384  1 ipt_REJECT
>>>>>>>>>>> xt_LOG                 20480  26
>>>>>>>>>>> xt_limit               16384  25
>>>>>>>>>>> xt_mark                16384  27
>>>>>>>>>>> xt_connmark            16384  2
>>>>>>>>>>> nf_log_syslog          24576  26
>>>>>>>>>>> iptable_raw            16384  0
>>>>>>>>>>> iptable_mangle         16384  1
>>>>>>>>>>> iptable_filter         16384  1
>>>>>>>>>>> vfat                   24576  1
>>>>>>>>>>> fat                    90112  1 vfat
>>>>>>>>>>> sch_cake               36864  4
>>>>>>>>>>> intel_powerclamp       20480  0
>>>>>>>>>>> psmouse               184320  0
>>>>>>>>>>> pcspkr                 16384  0
>>>>>>>>>>> i2c_piix4              28672  0
>>>>>>>>>>> e1000                 163840  0
>>>>>>>>>>> i2c_core              106496  2 psmouse,i2c_piix4
>>>>>>>>>>> crct10dif_pclmul       16384  1
>>>>>>>>>>> crc32_pclmul           16384  0
>>>>>>>>>>> ata_generic            16384  0
>>>>>>>>>>> pata_acpi              16384  0
>>>>>>>>>>> ghash_clmulni_intel    16384  0
>>>>>>>>>>> serio_raw              20480  0
>>>>>>>>>>> ohci_pci               20480  0
>>>>>>>>>>> video                  57344  0
>>>>>>>>>>
>>>>>>>>>> Yes, this looks more like it.
>>>>>>>>>>
>>>>>>>>>>> The last 8 entries are the same but a whole load of others are missing.
>>>>>>>>>>
>>>>>>>>>> Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
>>>>>>>>>>
>>>>>>>>>> I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
>>>>>>>>>>
>>>>>>>>>> -Michael
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Adolf.
>>>>>>>>>>>>> Tried modprobe e1000 but this came back with the following error
>>>>>>>>>>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service
>>>>>>>>>>>>>
>>>>>>>>>>>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards
>>>>>>>>>>>>> Adolf
>>>>>>>>>>>>>>> On 11 May 2022, at 20:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 11/05/2022 20:48, Jon Murphy wrote:
>>>>>>>>>>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
>>>>>>>>>>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c*
>>>>>>>>>>>>>>>> Shouldn’t this be Core Update 168?
>>>>>>>>>>>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>>>> That seems to be a bug that should not be there. Did we recently update apache?
>>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think I am going to raise a bug on this.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka(a)ipfire.org <mailto:adolf.belka(a)ipfire.org>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote:
>>>>>>>>>>>>>>>>>> Hi Leo,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote:
>>>>>>>>>>>>>>>>>>> Hi Adolf,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
>>>>>>>>>>>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
>>>>>>>>>>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 <https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762>
>>>>>>>>>>>>>>>>>> Now I see it in the code. Thanks very much.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
>>>>>>>>>>>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
>>>>>>>>>>>>>>>>>> Will wait to hear other inputs on this problem.
>>>>>>>>>>>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>>>>>> Leo
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>>>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> First try:-
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> starting
>>>>>>>>>>>>>>>>>>>> DUID 00:04:70:........
>>>>>>>>>>>>>>>>>>>> red0: interface not found
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Loading of module with unavailable key is rejected
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Second try:-
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> This time running the update took over 10 mins.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Repository was set back to Stable.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Before running the pakfire directory had the following
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> After running it had
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Rebooted.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Third try:-
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>>>>>>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>>>>>>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>>>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>>>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>>>>>>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>>>>>>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>>>>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Rebooted
>>>>>>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>>>>>>> Several repeats of the following two lines on the console screen:-
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>>>>>>>>>>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I have given up at this point.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>
> 

Re: Feedback on problems with Core Update 168 Testing

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 32600 bytes --]

Hello,

That would be brilliant. Let me know how it is going.

You would need to run this first:

https://patchwork.ipfire.org/project/ipfire/patch/20220519085634.197389-1-michael.tremer(a)ipfire.org/

And then just run the script and reboot. The boot process might stall for a minute. Just let it do its job. And the RAID should come up just fine.

The script will also avoid the previous issues with the file system that you have outlined after an unsuccessful resync.

-Michael

> On 19 May 2022, at 12:21, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi,
> 
> On 19/05/2022 13:14, Michael Tremer wrote:
>> Hello,
>>> On 19 May 2022, at 11:57, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>> 
>>> Hi Michael,
>>> 
>>> On 19/05/2022 10:59, Michael Tremer wrote:
>>>> Please note the patch that I just posted to fix any broken RAID arrays.
>>>> Does anyone have any systems that we can use to test this?
>>> Does the script need to be run manually or has it been added to the nightlies together with the rd.auto patch so that I can do a CU168 Testing update evaluation?
>> My patch includes that it will run automatically with the updater. However, that is not part of the build yet, because I would like it to be confirmed that I do not destroy anything. The whole script is a rather risky operation :)
> Okay, clear.
> Then I will execute the rd.auto change reboot and then run the script.
> 
> Will feedback how things go.
> Adolf.
>>> 
>>> Regards,
>>> Adolf.
>>> 
>>>> -Michael
>>>>> On 17 May 2022, at 09:09, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>> 
>>>>> Hi Michael,
>>>>> 
>>>>> On 13/05/2022 22:37, Michael Tremer wrote:
>>>>>> Hello,
>>>>>> Okay, this is really bad then if we are breaking RAID installations.
>>>>>> I tried to debug this, but VirtualBox is not my friend today.
>>>>>> I asked some bug reporter to provide me with the logs that I want over here:
>>>>>>   https://bugzilla.ipfire.org/show_bug.cgi?id=12862
>>>>> After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
>>>>> 
>>>>> My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
>>>>> 
>>>>> Thanks very much for all the help.
>>>>> 
>>>>> Regards,
>>>>> Adolf.
>>>>>> -Michael
>>>>>>> On 13 May 2022, at 18:12, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>> 
>>>>>>> Hi All,
>>>>>>> 
>>>>>>> I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
>>>>>>> 
>>>>>>> I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
>>>>>>> 
>>>>>>> I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
>>>>>>> 
>>>>>>> The vm shows the following
>>>>>>> 
>>>>>>> -bash-5.1$ cat /proc/mdstat
>>>>>>> Personalities :
>>>>>>> md127 : inactive sdb[1](S)
>>>>>>>       52428724 blocks super 1.0
>>>>>>> 
>>>>>>> unused devices: <none>
>>>>>>> 
>>>>>>> 
>>>>>>> It looks like IPFire was only running on sda.
>>>>>>> 
>>>>>>> So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
>>>>>>> 
>>>>>>> The CU168 vm that is working has the same as the above setting.
>>>>>>> 
>>>>>>> The CU168 vm that is not working has the following setting
>>>>>>> 
>>>>>>> -bash-5.1$ cat /proc/mdstat
>>>>>>> Personalities :
>>>>>>> md127 : inactive sda[0](S)
>>>>>>>       52428724 blocks super 1.0
>>>>>>> 
>>>>>>> unused devices: <none>
>>>>>>> 
>>>>>>> So it looks like it has booted up on the disk that was not updated from the raid pair.
>>>>>>> 
>>>>>>> 
>>>>>>> This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
>>>>>>> 
>>>>>>> It is probably dependent on which of the two disks actually get used to boot from.
>>>>>>> 
>>>>>>> So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
>>>>>>> 
>>>>>>> So raid systems need to have their raid arrays fully working before doing an upgrade.
>>>>>>> 
>>>>>>> At least the above is my interpretation of what I have found. What does everyone else think.
>>>>>>> 
>>>>>>> 
>>>>>>> Regards,
>>>>>>> 
>>>>>>> Adolf.
>>>>>>> 
>>>>>>> 
>>>>>>> On 13/05/2022 15:59, Adolf Belka wrote:
>>>>>>>> Hi Michael,
>>>>>>>> 
>>>>>>>> I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
>>>>>>>> 
>>>>>>>> I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
>>>>>>>> 
>>>>>>>> However I did copy the boot directory info for these two cases and found an interesting effect.
>>>>>>>> 
>>>>>>>> For the problem vm here was the boot directory layout of the CU167 base machine
>>>>>>>> 
>>>>>>>> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
>>>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>>>>> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>>>>>>> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
>>>>>>>> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
>>>>>>>> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>>>>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>>>>>> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>>>>>>> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>>>>>> 
>>>>>>>> After upgrade and before reboot this had changed to
>>>>>>>> 
>>>>>>>> drwxr-xr-x  5 root root 4.0K May 13 15:40 .
>>>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>>>>> -rw-r--r--  1 root root 179K May  3 04:02 config-5.15.35-ipfire
>>>>>>>> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
>>>>>>>> drwxr-xr-x  6 root root 4.0K May 13 15:41 grub
>>>>>>>> -rw-------  1 root root  26M May 13 15:41 initramfs-5.15.35-ipfire.img
>>>>>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>>>>>> -rw-r--r--  1 root root 4.7M May  3 04:02 System.map-5.15.35-ipfire
>>>>>>>> -rw-r--r--  1 root root 6.7M May  3 04:02 vmlinuz-5.15.35-ipfire
>>>>>>>> 
>>>>>>>> Dates and times of some of the files/directories have changed as would be expected.
>>>>>>>> 
>>>>>>>> After Reboot and with the problem the boot directory was now
>>>>>>>> 
>>>>>>>> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
>>>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>>>>> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>>>>>>> drwxr-xr-x  3 root root 4.0K Sep 30  2021 efi
>>>>>>>> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
>>>>>>>> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>>>>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>>>>>> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>>>>>>> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>>>>>> 
>>>>>>>> Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
>>>>>>>> 
>>>>>>>> For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
>>>>>>>> 
>>>>>>>> I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> 
>>>>>>>> Adolf.
>>>>>>>> 
>>>>>>>> On 13/05/2022 12:09, Michael Tremer wrote:
>>>>>>>>> Hello,
>>>>>>>>> 
>>>>>>>>>> On 12 May 2022, at 21:10, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi Michael,
>>>>>>>>>> 
>>>>>>>>>> On 12/05/2022 14:53, Michael Tremer wrote:
>>>>>>>>>>> Hello,
>>>>>>>>>>>> On 12 May 2022, at 12:25, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> 
>>>>>>>>>>>> On 12/05/2022 11:13, Michael Tremer wrote:
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
>>>>>>>>>>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
>>>>>>>>>>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
>>>>>>>>>>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
>>>>>>>>>>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
>>>>>>>>>>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
>>>>>>>>>>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
>>>>>>>>>>> Okay. That is indeed quite bad then.
>>>>>>>>>>> Since there is no kernel in 168, is there a chance that we broke the update to 167?
>>>>>>>>>> I don't know but as far as I have been concerned CU167 has worked well on my vm.
>>>>>>>>>> Is there something I should look for on my CU167 vm?
>>>>>>>>>>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
>>>>>>>>>>> Third time lucky isn’t good enough for me :)
>>>>>>>>>> Me neither :-)
>>>>>>>>>>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
>>>>>>>>>>>> 
>>>>>>>>>>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
>>>>>>>>>>> Have there been any modules loaded? If one loads, they all should load.
>>>>>>>>>> This is what I get with lsmod
>>>>>>>>>> 
>>>>>>>>>> Module            Size    Used by
>>>>>>>>>> crct10dif_pclmul    16384    1
>>>>>>>>>> crc32_pclmul        16384    0
>>>>>>>>>> ghash_clmulni_intel    16384    0
>>>>>>>>>> serio_raw        20480    0
>>>>>>>>>> ohci_pci        20480    0
>>>>>>>>>> ata_generic        16384    0
>>>>>>>>>> pata_acpi        16384    0
>>>>>>>>>> video            57344    0
>>>>>>>>> 
>>>>>>>>> It looks like these modules could have come from the initramdisk image. At least that is working well.
>>>>>>>>> 
>>>>>>>>>> On my running CU1678 vm lsmod gives the following
>>>>>>>>>> 
>>>>>>>>>> Module                  Size  Used by
>>>>>>>>>> tun                    61440  2
>>>>>>>>>> nfnetlink_queue        28672  1
>>>>>>>>>> xt_NFQUEUE             16384  8
>>>>>>>>>> xt_MASQUERADE          20480  1
>>>>>>>>>> cfg80211             1036288  0
>>>>>>>>>> rfkill                 32768  1 cfg80211
>>>>>>>>>> 8021q                  40960  0
>>>>>>>>>> garp                   16384  1 8021q
>>>>>>>>>> xt_set                 16384  260
>>>>>>>>>> ip_set_hash_net        49152  258
>>>>>>>>>> ip_set                 57344  2 xt_set,ip_set_hash_net
>>>>>>>>>> xt_hashlimit           20480  2
>>>>>>>>>> xt_multiport           20480  4
>>>>>>>>>> xt_policy              16384  5
>>>>>>>>>> xt_TCPMSS              16384  1
>>>>>>>>>> xt_conntrack           16384  7
>>>>>>>>>> xt_comment             16384  18
>>>>>>>>>> ipt_REJECT             16384  1
>>>>>>>>>> nf_reject_ipv4         16384  1 ipt_REJECT
>>>>>>>>>> xt_LOG                 20480  26
>>>>>>>>>> xt_limit               16384  25
>>>>>>>>>> xt_mark                16384  27
>>>>>>>>>> xt_connmark            16384  2
>>>>>>>>>> nf_log_syslog          24576  26
>>>>>>>>>> iptable_raw            16384  0
>>>>>>>>>> iptable_mangle         16384  1
>>>>>>>>>> iptable_filter         16384  1
>>>>>>>>>> vfat                   24576  1
>>>>>>>>>> fat                    90112  1 vfat
>>>>>>>>>> sch_cake               36864  4
>>>>>>>>>> intel_powerclamp       20480  0
>>>>>>>>>> psmouse               184320  0
>>>>>>>>>> pcspkr                 16384  0
>>>>>>>>>> i2c_piix4              28672  0
>>>>>>>>>> e1000                 163840  0
>>>>>>>>>> i2c_core              106496  2 psmouse,i2c_piix4
>>>>>>>>>> crct10dif_pclmul       16384  1
>>>>>>>>>> crc32_pclmul           16384  0
>>>>>>>>>> ata_generic            16384  0
>>>>>>>>>> pata_acpi              16384  0
>>>>>>>>>> ghash_clmulni_intel    16384  0
>>>>>>>>>> serio_raw              20480  0
>>>>>>>>>> ohci_pci               20480  0
>>>>>>>>>> video                  57344  0
>>>>>>>>> 
>>>>>>>>> Yes, this looks more like it.
>>>>>>>>> 
>>>>>>>>>> The last 8 entries are the same but a whole load of others are missing.
>>>>>>>>> 
>>>>>>>>> Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
>>>>>>>>> 
>>>>>>>>> I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
>>>>>>>>> 
>>>>>>>>> -Michael
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> Adolf.
>>>>>>>>>>>> Tried modprobe e1000 but this came back with the following error
>>>>>>>>>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service
>>>>>>>>>>>> 
>>>>>>>>>>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
>>>>>>>>>>>> 
>>>>>>>>>>>> Regards
>>>>>>>>>>>> Adolf
>>>>>>>>>>>>>> On 11 May 2022, at 20:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 11/05/2022 20:48, Jon Murphy wrote:
>>>>>>>>>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
>>>>>>>>>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c*
>>>>>>>>>>>>>>> Shouldn’t this be Core Update 168?
>>>>>>>>>>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>>> That seems to be a bug that should not be there. Did we recently update apache?
>>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I think I am going to raise a bug on this.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka(a)ipfire.org <mailto:adolf.belka(a)ipfire.org>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote:
>>>>>>>>>>>>>>>>> Hi Leo,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote:
>>>>>>>>>>>>>>>>>> Hi Adolf,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
>>>>>>>>>>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
>>>>>>>>>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 <https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762>
>>>>>>>>>>>>>>>>> Now I see it in the code. Thanks very much.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
>>>>>>>>>>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
>>>>>>>>>>>>>>>>> Will wait to hear other inputs on this problem.
>>>>>>>>>>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>>>>> Leo
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> First try:-
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> starting
>>>>>>>>>>>>>>>>>>> DUID 00:04:70:........
>>>>>>>>>>>>>>>>>>> red0: interface not found
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Loading of module with unavailable key is rejected
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Second try:-
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> This time running the update took over 10 mins.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Repository was set back to Stable.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Before running the pakfire directory had the following
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> After running it had
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Rebooted.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Third try:-
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>>>>>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>>>>>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>>>>>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>>>>>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>>>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Rebooted
>>>>>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>>>>>> Several repeats of the following two lines on the console screen:-
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>>>>>>>>>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I have given up at this point.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>> 

Re: Feedback on problems with Core Update 168 Testing

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 31385 bytes --]

Hi,

On 19/05/2022 13:14, Michael Tremer wrote:
> Hello,
> 
>> On 19 May 2022, at 11:57, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi Michael,
>>
>> On 19/05/2022 10:59, Michael Tremer wrote:
>>> Please note the patch that I just posted to fix any broken RAID arrays.
>>> Does anyone have any systems that we can use to test this?
>> Does the script need to be run manually or has it been added to the nightlies together with the rd.auto patch so that I can do a CU168 Testing update evaluation?
> 
> My patch includes that it will run automatically with the updater. However, that is not part of the build yet, because I would like it to be confirmed that I do not destroy anything. The whole script is a rather risky operation :)
Okay, clear.
Then I will execute the rd.auto change reboot and then run the script.

Will feedback how things go.
Adolf.
> 
>>
>> Regards,
>> Adolf.
>>
>>> -Michael
>>>> On 17 May 2022, at 09:09, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>
>>>> Hi Michael,
>>>>
>>>> On 13/05/2022 22:37, Michael Tremer wrote:
>>>>> Hello,
>>>>> Okay, this is really bad then if we are breaking RAID installations.
>>>>> I tried to debug this, but VirtualBox is not my friend today.
>>>>> I asked some bug reporter to provide me with the logs that I want over here:
>>>>>    https://bugzilla.ipfire.org/show_bug.cgi?id=12862
>>>> After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
>>>>
>>>> My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
>>>>
>>>> Thanks very much for all the help.
>>>>
>>>> Regards,
>>>> Adolf.
>>>>> -Michael
>>>>>> On 13 May 2022, at 18:12, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
>>>>>>
>>>>>> I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
>>>>>>
>>>>>> I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
>>>>>>
>>>>>> The vm shows the following
>>>>>>
>>>>>> -bash-5.1$ cat /proc/mdstat
>>>>>> Personalities :
>>>>>> md127 : inactive sdb[1](S)
>>>>>>        52428724 blocks super 1.0
>>>>>>
>>>>>> unused devices: <none>
>>>>>>
>>>>>>
>>>>>> It looks like IPFire was only running on sda.
>>>>>>
>>>>>> So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
>>>>>>
>>>>>> The CU168 vm that is working has the same as the above setting.
>>>>>>
>>>>>> The CU168 vm that is not working has the following setting
>>>>>>
>>>>>> -bash-5.1$ cat /proc/mdstat
>>>>>> Personalities :
>>>>>> md127 : inactive sda[0](S)
>>>>>>        52428724 blocks super 1.0
>>>>>>
>>>>>> unused devices: <none>
>>>>>>
>>>>>> So it looks like it has booted up on the disk that was not updated from the raid pair.
>>>>>>
>>>>>>
>>>>>> This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
>>>>>>
>>>>>> It is probably dependent on which of the two disks actually get used to boot from.
>>>>>>
>>>>>> So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
>>>>>>
>>>>>> So raid systems need to have their raid arrays fully working before doing an upgrade.
>>>>>>
>>>>>> At least the above is my interpretation of what I have found. What does everyone else think.
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Adolf.
>>>>>>
>>>>>>
>>>>>> On 13/05/2022 15:59, Adolf Belka wrote:
>>>>>>> Hi Michael,
>>>>>>>
>>>>>>> I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
>>>>>>>
>>>>>>> I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
>>>>>>>
>>>>>>> However I did copy the boot directory info for these two cases and found an interesting effect.
>>>>>>>
>>>>>>> For the problem vm here was the boot directory layout of the CU167 base machine
>>>>>>>
>>>>>>> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
>>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>>>> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>>>>>> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
>>>>>>> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
>>>>>>> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>>>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>>>>> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>>>>>> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>>>>>
>>>>>>> After upgrade and before reboot this had changed to
>>>>>>>
>>>>>>> drwxr-xr-x  5 root root 4.0K May 13 15:40 .
>>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>>>> -rw-r--r--  1 root root 179K May  3 04:02 config-5.15.35-ipfire
>>>>>>> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
>>>>>>> drwxr-xr-x  6 root root 4.0K May 13 15:41 grub
>>>>>>> -rw-------  1 root root  26M May 13 15:41 initramfs-5.15.35-ipfire.img
>>>>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>>>>> -rw-r--r--  1 root root 4.7M May  3 04:02 System.map-5.15.35-ipfire
>>>>>>> -rw-r--r--  1 root root 6.7M May  3 04:02 vmlinuz-5.15.35-ipfire
>>>>>>>
>>>>>>> Dates and times of some of the files/directories have changed as would be expected.
>>>>>>>
>>>>>>> After Reboot and with the problem the boot directory was now
>>>>>>>
>>>>>>> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
>>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>>>> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>>>>>> drwxr-xr-x  3 root root 4.0K Sep 30  2021 efi
>>>>>>> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
>>>>>>> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>>>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>>>>> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>>>>>> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>>>>>
>>>>>>> Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
>>>>>>>
>>>>>>> For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
>>>>>>>
>>>>>>> I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Adolf.
>>>>>>>
>>>>>>> On 13/05/2022 12:09, Michael Tremer wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>>> On 12 May 2022, at 21:10, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>
>>>>>>>>> Hi Michael,
>>>>>>>>>
>>>>>>>>> On 12/05/2022 14:53, Michael Tremer wrote:
>>>>>>>>>> Hello,
>>>>>>>>>>> On 12 May 2022, at 12:25, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> On 12/05/2022 11:13, Michael Tremer wrote:
>>>>>>>>>>>> Hello,
>>>>>>>>>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
>>>>>>>>>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
>>>>>>>>>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
>>>>>>>>>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
>>>>>>>>>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
>>>>>>>>>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
>>>>>>>>>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
>>>>>>>>>> Okay. That is indeed quite bad then.
>>>>>>>>>> Since there is no kernel in 168, is there a chance that we broke the update to 167?
>>>>>>>>> I don't know but as far as I have been concerned CU167 has worked well on my vm.
>>>>>>>>> Is there something I should look for on my CU167 vm?
>>>>>>>>>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
>>>>>>>>>> Third time lucky isn’t good enough for me :)
>>>>>>>>> Me neither :-)
>>>>>>>>>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
>>>>>>>>>>>
>>>>>>>>>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
>>>>>>>>>> Have there been any modules loaded? If one loads, they all should load.
>>>>>>>>> This is what I get with lsmod
>>>>>>>>>
>>>>>>>>> Module            Size    Used by
>>>>>>>>> crct10dif_pclmul    16384    1
>>>>>>>>> crc32_pclmul        16384    0
>>>>>>>>> ghash_clmulni_intel    16384    0
>>>>>>>>> serio_raw        20480    0
>>>>>>>>> ohci_pci        20480    0
>>>>>>>>> ata_generic        16384    0
>>>>>>>>> pata_acpi        16384    0
>>>>>>>>> video            57344    0
>>>>>>>>
>>>>>>>> It looks like these modules could have come from the initramdisk image. At least that is working well.
>>>>>>>>
>>>>>>>>> On my running CU1678 vm lsmod gives the following
>>>>>>>>>
>>>>>>>>> Module                  Size  Used by
>>>>>>>>> tun                    61440  2
>>>>>>>>> nfnetlink_queue        28672  1
>>>>>>>>> xt_NFQUEUE             16384  8
>>>>>>>>> xt_MASQUERADE          20480  1
>>>>>>>>> cfg80211             1036288  0
>>>>>>>>> rfkill                 32768  1 cfg80211
>>>>>>>>> 8021q                  40960  0
>>>>>>>>> garp                   16384  1 8021q
>>>>>>>>> xt_set                 16384  260
>>>>>>>>> ip_set_hash_net        49152  258
>>>>>>>>> ip_set                 57344  2 xt_set,ip_set_hash_net
>>>>>>>>> xt_hashlimit           20480  2
>>>>>>>>> xt_multiport           20480  4
>>>>>>>>> xt_policy              16384  5
>>>>>>>>> xt_TCPMSS              16384  1
>>>>>>>>> xt_conntrack           16384  7
>>>>>>>>> xt_comment             16384  18
>>>>>>>>> ipt_REJECT             16384  1
>>>>>>>>> nf_reject_ipv4         16384  1 ipt_REJECT
>>>>>>>>> xt_LOG                 20480  26
>>>>>>>>> xt_limit               16384  25
>>>>>>>>> xt_mark                16384  27
>>>>>>>>> xt_connmark            16384  2
>>>>>>>>> nf_log_syslog          24576  26
>>>>>>>>> iptable_raw            16384  0
>>>>>>>>> iptable_mangle         16384  1
>>>>>>>>> iptable_filter         16384  1
>>>>>>>>> vfat                   24576  1
>>>>>>>>> fat                    90112  1 vfat
>>>>>>>>> sch_cake               36864  4
>>>>>>>>> intel_powerclamp       20480  0
>>>>>>>>> psmouse               184320  0
>>>>>>>>> pcspkr                 16384  0
>>>>>>>>> i2c_piix4              28672  0
>>>>>>>>> e1000                 163840  0
>>>>>>>>> i2c_core              106496  2 psmouse,i2c_piix4
>>>>>>>>> crct10dif_pclmul       16384  1
>>>>>>>>> crc32_pclmul           16384  0
>>>>>>>>> ata_generic            16384  0
>>>>>>>>> pata_acpi              16384  0
>>>>>>>>> ghash_clmulni_intel    16384  0
>>>>>>>>> serio_raw              20480  0
>>>>>>>>> ohci_pci               20480  0
>>>>>>>>> video                  57344  0
>>>>>>>>
>>>>>>>> Yes, this looks more like it.
>>>>>>>>
>>>>>>>>> The last 8 entries are the same but a whole load of others are missing.
>>>>>>>>
>>>>>>>> Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
>>>>>>>>
>>>>>>>> I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
>>>>>>>>
>>>>>>>> -Michael
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Adolf.
>>>>>>>>>>> Tried modprobe e1000 but this came back with the following error
>>>>>>>>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service
>>>>>>>>>>>
>>>>>>>>>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
>>>>>>>>>>>
>>>>>>>>>>> Regards
>>>>>>>>>>> Adolf
>>>>>>>>>>>>> On 11 May 2022, at 20:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 11/05/2022 20:48, Jon Murphy wrote:
>>>>>>>>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
>>>>>>>>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c*
>>>>>>>>>>>>>> Shouldn’t this be Core Update 168?
>>>>>>>>>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>> That seems to be a bug that should not be there. Did we recently update apache?
>>>>>>>>>>>> -Michael
>>>>>>>>>>>>>
>>>>>>>>>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think I am going to raise a bug on this.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka(a)ipfire.org <mailto:adolf.belka(a)ipfire.org>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote:
>>>>>>>>>>>>>>>> Hi Leo,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote:
>>>>>>>>>>>>>>>>> Hi Adolf,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
>>>>>>>>>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
>>>>>>>>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 <https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762>
>>>>>>>>>>>>>>>> Now I see it in the code. Thanks very much.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
>>>>>>>>>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
>>>>>>>>>>>>>>>> Will wait to hear other inputs on this problem.
>>>>>>>>>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>>>> Leo
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> First try:-
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> starting
>>>>>>>>>>>>>>>>>> DUID 00:04:70:........
>>>>>>>>>>>>>>>>>> red0: interface not found
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Loading of module with unavailable key is rejected
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Second try:-
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This time running the update took over 10 mins.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Repository was set back to Stable.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Before running the pakfire directory had the following
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> After running it had
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Rebooted.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Third try:-
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>>>>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>>>>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>>>>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>>>>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Rebooted
>>>>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>>>>> Several repeats of the following two lines on the console screen:-
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>>>>>>>>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I have given up at this point.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Adolf.
>>>>>>>>
> 

Re: Feedback on problems with Core Update 168 Testing

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 30848 bytes --]

Hello,

> On 19 May 2022, at 11:57, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Michael,
> 
> On 19/05/2022 10:59, Michael Tremer wrote:
>> Please note the patch that I just posted to fix any broken RAID arrays.
>> Does anyone have any systems that we can use to test this?
> Does the script need to be run manually or has it been added to the nightlies together with the rd.auto patch so that I can do a CU168 Testing update evaluation?

My patch includes that it will run automatically with the updater. However, that is not part of the build yet, because I would like it to be confirmed that I do not destroy anything. The whole script is a rather risky operation :)

> 
> Regards,
> Adolf.
> 
>> -Michael
>>> On 17 May 2022, at 09:09, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>> 
>>> Hi Michael,
>>> 
>>> On 13/05/2022 22:37, Michael Tremer wrote:
>>>> Hello,
>>>> Okay, this is really bad then if we are breaking RAID installations.
>>>> I tried to debug this, but VirtualBox is not my friend today.
>>>> I asked some bug reporter to provide me with the logs that I want over here:
>>>>   https://bugzilla.ipfire.org/show_bug.cgi?id=12862
>>> After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
>>> 
>>> My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
>>> 
>>> Thanks very much for all the help.
>>> 
>>> Regards,
>>> Adolf.
>>>> -Michael
>>>>> On 13 May 2022, at 18:12, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>> 
>>>>> Hi All,
>>>>> 
>>>>> I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
>>>>> 
>>>>> I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
>>>>> 
>>>>> I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
>>>>> 
>>>>> The vm shows the following
>>>>> 
>>>>> -bash-5.1$ cat /proc/mdstat
>>>>> Personalities :
>>>>> md127 : inactive sdb[1](S)
>>>>>       52428724 blocks super 1.0
>>>>> 
>>>>> unused devices: <none>
>>>>> 
>>>>> 
>>>>> It looks like IPFire was only running on sda.
>>>>> 
>>>>> So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
>>>>> 
>>>>> The CU168 vm that is working has the same as the above setting.
>>>>> 
>>>>> The CU168 vm that is not working has the following setting
>>>>> 
>>>>> -bash-5.1$ cat /proc/mdstat
>>>>> Personalities :
>>>>> md127 : inactive sda[0](S)
>>>>>       52428724 blocks super 1.0
>>>>> 
>>>>> unused devices: <none>
>>>>> 
>>>>> So it looks like it has booted up on the disk that was not updated from the raid pair.
>>>>> 
>>>>> 
>>>>> This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
>>>>> 
>>>>> It is probably dependent on which of the two disks actually get used to boot from.
>>>>> 
>>>>> So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
>>>>> 
>>>>> So raid systems need to have their raid arrays fully working before doing an upgrade.
>>>>> 
>>>>> At least the above is my interpretation of what I have found. What does everyone else think.
>>>>> 
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> Adolf.
>>>>> 
>>>>> 
>>>>> On 13/05/2022 15:59, Adolf Belka wrote:
>>>>>> Hi Michael,
>>>>>> 
>>>>>> I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
>>>>>> 
>>>>>> I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
>>>>>> 
>>>>>> However I did copy the boot directory info for these two cases and found an interesting effect.
>>>>>> 
>>>>>> For the problem vm here was the boot directory layout of the CU167 base machine
>>>>>> 
>>>>>> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>>> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>>>>> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
>>>>>> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
>>>>>> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>>>> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>>>>> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>>>> 
>>>>>> After upgrade and before reboot this had changed to
>>>>>> 
>>>>>> drwxr-xr-x  5 root root 4.0K May 13 15:40 .
>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>>> -rw-r--r--  1 root root 179K May  3 04:02 config-5.15.35-ipfire
>>>>>> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
>>>>>> drwxr-xr-x  6 root root 4.0K May 13 15:41 grub
>>>>>> -rw-------  1 root root  26M May 13 15:41 initramfs-5.15.35-ipfire.img
>>>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>>>> -rw-r--r--  1 root root 4.7M May  3 04:02 System.map-5.15.35-ipfire
>>>>>> -rw-r--r--  1 root root 6.7M May  3 04:02 vmlinuz-5.15.35-ipfire
>>>>>> 
>>>>>> Dates and times of some of the files/directories have changed as would be expected.
>>>>>> 
>>>>>> After Reboot and with the problem the boot directory was now
>>>>>> 
>>>>>> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
>>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>>> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>>>>> drwxr-xr-x  3 root root 4.0K Sep 30  2021 efi
>>>>>> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
>>>>>> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>>>> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>>>>> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>>>> 
>>>>>> Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
>>>>>> 
>>>>>> For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
>>>>>> 
>>>>>> I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
>>>>>> 
>>>>>> Regards,
>>>>>> 
>>>>>> Adolf.
>>>>>> 
>>>>>> On 13/05/2022 12:09, Michael Tremer wrote:
>>>>>>> Hello,
>>>>>>> 
>>>>>>>> On 12 May 2022, at 21:10, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>> 
>>>>>>>> Hi Michael,
>>>>>>>> 
>>>>>>>> On 12/05/2022 14:53, Michael Tremer wrote:
>>>>>>>>> Hello,
>>>>>>>>>> On 12 May 2022, at 12:25, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi,
>>>>>>>>>> 
>>>>>>>>>> On 12/05/2022 11:13, Michael Tremer wrote:
>>>>>>>>>>> Hello,
>>>>>>>>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
>>>>>>>>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
>>>>>>>>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
>>>>>>>>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
>>>>>>>>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
>>>>>>>>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
>>>>>>>>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
>>>>>>>>> Okay. That is indeed quite bad then.
>>>>>>>>> Since there is no kernel in 168, is there a chance that we broke the update to 167?
>>>>>>>> I don't know but as far as I have been concerned CU167 has worked well on my vm.
>>>>>>>> Is there something I should look for on my CU167 vm?
>>>>>>>>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
>>>>>>>>> Third time lucky isn’t good enough for me :)
>>>>>>>> Me neither :-)
>>>>>>>>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
>>>>>>>>>> 
>>>>>>>>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
>>>>>>>>> Have there been any modules loaded? If one loads, they all should load.
>>>>>>>> This is what I get with lsmod
>>>>>>>> 
>>>>>>>> Module            Size    Used by
>>>>>>>> crct10dif_pclmul    16384    1
>>>>>>>> crc32_pclmul        16384    0
>>>>>>>> ghash_clmulni_intel    16384    0
>>>>>>>> serio_raw        20480    0
>>>>>>>> ohci_pci        20480    0
>>>>>>>> ata_generic        16384    0
>>>>>>>> pata_acpi        16384    0
>>>>>>>> video            57344    0
>>>>>>> 
>>>>>>> It looks like these modules could have come from the initramdisk image. At least that is working well.
>>>>>>> 
>>>>>>>> On my running CU1678 vm lsmod gives the following
>>>>>>>> 
>>>>>>>> Module                  Size  Used by
>>>>>>>> tun                    61440  2
>>>>>>>> nfnetlink_queue        28672  1
>>>>>>>> xt_NFQUEUE             16384  8
>>>>>>>> xt_MASQUERADE          20480  1
>>>>>>>> cfg80211             1036288  0
>>>>>>>> rfkill                 32768  1 cfg80211
>>>>>>>> 8021q                  40960  0
>>>>>>>> garp                   16384  1 8021q
>>>>>>>> xt_set                 16384  260
>>>>>>>> ip_set_hash_net        49152  258
>>>>>>>> ip_set                 57344  2 xt_set,ip_set_hash_net
>>>>>>>> xt_hashlimit           20480  2
>>>>>>>> xt_multiport           20480  4
>>>>>>>> xt_policy              16384  5
>>>>>>>> xt_TCPMSS              16384  1
>>>>>>>> xt_conntrack           16384  7
>>>>>>>> xt_comment             16384  18
>>>>>>>> ipt_REJECT             16384  1
>>>>>>>> nf_reject_ipv4         16384  1 ipt_REJECT
>>>>>>>> xt_LOG                 20480  26
>>>>>>>> xt_limit               16384  25
>>>>>>>> xt_mark                16384  27
>>>>>>>> xt_connmark            16384  2
>>>>>>>> nf_log_syslog          24576  26
>>>>>>>> iptable_raw            16384  0
>>>>>>>> iptable_mangle         16384  1
>>>>>>>> iptable_filter         16384  1
>>>>>>>> vfat                   24576  1
>>>>>>>> fat                    90112  1 vfat
>>>>>>>> sch_cake               36864  4
>>>>>>>> intel_powerclamp       20480  0
>>>>>>>> psmouse               184320  0
>>>>>>>> pcspkr                 16384  0
>>>>>>>> i2c_piix4              28672  0
>>>>>>>> e1000                 163840  0
>>>>>>>> i2c_core              106496  2 psmouse,i2c_piix4
>>>>>>>> crct10dif_pclmul       16384  1
>>>>>>>> crc32_pclmul           16384  0
>>>>>>>> ata_generic            16384  0
>>>>>>>> pata_acpi              16384  0
>>>>>>>> ghash_clmulni_intel    16384  0
>>>>>>>> serio_raw              20480  0
>>>>>>>> ohci_pci               20480  0
>>>>>>>> video                  57344  0
>>>>>>> 
>>>>>>> Yes, this looks more like it.
>>>>>>> 
>>>>>>>> The last 8 entries are the same but a whole load of others are missing.
>>>>>>> 
>>>>>>> Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
>>>>>>> 
>>>>>>> I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
>>>>>>> 
>>>>>>> -Michael
>>>>>>> 
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> Adolf.
>>>>>>>>>> Tried modprobe e1000 but this came back with the following error
>>>>>>>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service
>>>>>>>>>> 
>>>>>>>>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
>>>>>>>>>> 
>>>>>>>>>> Regards
>>>>>>>>>> Adolf
>>>>>>>>>>>> On 11 May 2022, at 20:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>> 
>>>>>>>>>>>> On 11/05/2022 20:48, Jon Murphy wrote:
>>>>>>>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
>>>>>>>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c*
>>>>>>>>>>>>> Shouldn’t this be Core Update 168?
>>>>>>>>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
>>>>>>>>>>>> 
>>>>>>>>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
>>>>>>>>>>>> 
>>>>>>>>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>> 
>>>>>>>>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>> 
>>>>>>>>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>> That seems to be a bug that should not be there. Did we recently update apache?
>>>>>>>>>>> -Michael
>>>>>>>>>>>> 
>>>>>>>>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
>>>>>>>>>>>> 
>>>>>>>>>>>> I think I am going to raise a bug on this.
>>>>>>>>>>>> 
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka(a)ipfire.org <mailto:adolf.belka(a)ipfire.org>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote:
>>>>>>>>>>>>>>> Hi Leo,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote:
>>>>>>>>>>>>>>>> Hi Adolf,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
>>>>>>>>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
>>>>>>>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 <https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762>
>>>>>>>>>>>>>>> Now I see it in the code. Thanks very much.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
>>>>>>>>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
>>>>>>>>>>>>>>> Will wait to hear other inputs on this problem.
>>>>>>>>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>>> Leo
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> First try:-
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> starting
>>>>>>>>>>>>>>>>> DUID 00:04:70:........
>>>>>>>>>>>>>>>>> red0: interface not found
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Loading of module with unavailable key is rejected
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Second try:-
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> This time running the update took over 10 mins.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Repository was set back to Stable.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Before running the pakfire directory had the following
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> After running it had
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Rebooted.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Third try:-
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>>>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>>>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>>>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>>>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Rebooted
>>>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>>>> Several repeats of the following two lines on the console screen:-
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>>>>>>>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I have given up at this point.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Adolf.
>>>>>>> 

Re: Feedback on problems with Core Update 168 Testing

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 29899 bytes --]

Hi Michael,

On 19/05/2022 10:59, Michael Tremer wrote:
> Please note the patch that I just posted to fix any broken RAID arrays.
> 
> Does anyone have any systems that we can use to test this?
Does the script need to be run manually or has it been added to the nightlies together with the rd.auto patch so that I can do a CU168 Testing update evaluation?

Regards,
Adolf.

> 
> -Michael
> 
>> On 17 May 2022, at 09:09, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi Michael,
>>
>> On 13/05/2022 22:37, Michael Tremer wrote:
>>> Hello,
>>> Okay, this is really bad then if we are breaking RAID installations.
>>> I tried to debug this, but VirtualBox is not my friend today.
>>> I asked some bug reporter to provide me with the logs that I want over here:
>>>    https://bugzilla.ipfire.org/show_bug.cgi?id=12862
>> After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
>>
>> My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
>>
>> Thanks very much for all the help.
>>
>> Regards,
>> Adolf.
>>> -Michael
>>>> On 13 May 2022, at 18:12, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>
>>>> Hi All,
>>>>
>>>> I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
>>>>
>>>> I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
>>>>
>>>> I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
>>>>
>>>> The vm shows the following
>>>>
>>>> -bash-5.1$ cat /proc/mdstat
>>>> Personalities :
>>>> md127 : inactive sdb[1](S)
>>>>        52428724 blocks super 1.0
>>>>
>>>> unused devices: <none>
>>>>
>>>>
>>>> It looks like IPFire was only running on sda.
>>>>
>>>> So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
>>>>
>>>> The CU168 vm that is working has the same as the above setting.
>>>>
>>>> The CU168 vm that is not working has the following setting
>>>>
>>>> -bash-5.1$ cat /proc/mdstat
>>>> Personalities :
>>>> md127 : inactive sda[0](S)
>>>>        52428724 blocks super 1.0
>>>>
>>>> unused devices: <none>
>>>>
>>>> So it looks like it has booted up on the disk that was not updated from the raid pair.
>>>>
>>>>
>>>> This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
>>>>
>>>> It is probably dependent on which of the two disks actually get used to boot from.
>>>>
>>>> So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
>>>>
>>>> So raid systems need to have their raid arrays fully working before doing an upgrade.
>>>>
>>>> At least the above is my interpretation of what I have found. What does everyone else think.
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Adolf.
>>>>
>>>>
>>>> On 13/05/2022 15:59, Adolf Belka wrote:
>>>>> Hi Michael,
>>>>>
>>>>> I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
>>>>>
>>>>> I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
>>>>>
>>>>> However I did copy the boot directory info for these two cases and found an interesting effect.
>>>>>
>>>>> For the problem vm here was the boot directory layout of the CU167 base machine
>>>>>
>>>>> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>>>> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
>>>>> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
>>>>> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>>> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>>>> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>>>
>>>>> After upgrade and before reboot this had changed to
>>>>>
>>>>> drwxr-xr-x  5 root root 4.0K May 13 15:40 .
>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>> -rw-r--r--  1 root root 179K May  3 04:02 config-5.15.35-ipfire
>>>>> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
>>>>> drwxr-xr-x  6 root root 4.0K May 13 15:41 grub
>>>>> -rw-------  1 root root  26M May 13 15:41 initramfs-5.15.35-ipfire.img
>>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>>> -rw-r--r--  1 root root 4.7M May  3 04:02 System.map-5.15.35-ipfire
>>>>> -rw-r--r--  1 root root 6.7M May  3 04:02 vmlinuz-5.15.35-ipfire
>>>>>
>>>>> Dates and times of some of the files/directories have changed as would be expected.
>>>>>
>>>>> After Reboot and with the problem the boot directory was now
>>>>>
>>>>> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>>>> drwxr-xr-x  3 root root 4.0K Sep 30  2021 efi
>>>>> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
>>>>> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>>> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>>>> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>>>
>>>>> Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
>>>>>
>>>>> For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
>>>>>
>>>>> I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Adolf.
>>>>>
>>>>> On 13/05/2022 12:09, Michael Tremer wrote:
>>>>>> Hello,
>>>>>>
>>>>>>> On 12 May 2022, at 21:10, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>
>>>>>>> Hi Michael,
>>>>>>>
>>>>>>> On 12/05/2022 14:53, Michael Tremer wrote:
>>>>>>>> Hello,
>>>>>>>>> On 12 May 2022, at 12:25, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On 12/05/2022 11:13, Michael Tremer wrote:
>>>>>>>>>> Hello,
>>>>>>>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
>>>>>>>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
>>>>>>>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
>>>>>>>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
>>>>>>>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
>>>>>>>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
>>>>>>>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
>>>>>>>> Okay. That is indeed quite bad then.
>>>>>>>> Since there is no kernel in 168, is there a chance that we broke the update to 167?
>>>>>>> I don't know but as far as I have been concerned CU167 has worked well on my vm.
>>>>>>> Is there something I should look for on my CU167 vm?
>>>>>>>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
>>>>>>>> Third time lucky isn’t good enough for me :)
>>>>>>> Me neither :-)
>>>>>>>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
>>>>>>>>>
>>>>>>>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
>>>>>>>> Have there been any modules loaded? If one loads, they all should load.
>>>>>>> This is what I get with lsmod
>>>>>>>
>>>>>>> Module            Size    Used by
>>>>>>> crct10dif_pclmul    16384    1
>>>>>>> crc32_pclmul        16384    0
>>>>>>> ghash_clmulni_intel    16384    0
>>>>>>> serio_raw        20480    0
>>>>>>> ohci_pci        20480    0
>>>>>>> ata_generic        16384    0
>>>>>>> pata_acpi        16384    0
>>>>>>> video            57344    0
>>>>>>
>>>>>> It looks like these modules could have come from the initramdisk image. At least that is working well.
>>>>>>
>>>>>>> On my running CU1678 vm lsmod gives the following
>>>>>>>
>>>>>>> Module                  Size  Used by
>>>>>>> tun                    61440  2
>>>>>>> nfnetlink_queue        28672  1
>>>>>>> xt_NFQUEUE             16384  8
>>>>>>> xt_MASQUERADE          20480  1
>>>>>>> cfg80211             1036288  0
>>>>>>> rfkill                 32768  1 cfg80211
>>>>>>> 8021q                  40960  0
>>>>>>> garp                   16384  1 8021q
>>>>>>> xt_set                 16384  260
>>>>>>> ip_set_hash_net        49152  258
>>>>>>> ip_set                 57344  2 xt_set,ip_set_hash_net
>>>>>>> xt_hashlimit           20480  2
>>>>>>> xt_multiport           20480  4
>>>>>>> xt_policy              16384  5
>>>>>>> xt_TCPMSS              16384  1
>>>>>>> xt_conntrack           16384  7
>>>>>>> xt_comment             16384  18
>>>>>>> ipt_REJECT             16384  1
>>>>>>> nf_reject_ipv4         16384  1 ipt_REJECT
>>>>>>> xt_LOG                 20480  26
>>>>>>> xt_limit               16384  25
>>>>>>> xt_mark                16384  27
>>>>>>> xt_connmark            16384  2
>>>>>>> nf_log_syslog          24576  26
>>>>>>> iptable_raw            16384  0
>>>>>>> iptable_mangle         16384  1
>>>>>>> iptable_filter         16384  1
>>>>>>> vfat                   24576  1
>>>>>>> fat                    90112  1 vfat
>>>>>>> sch_cake               36864  4
>>>>>>> intel_powerclamp       20480  0
>>>>>>> psmouse               184320  0
>>>>>>> pcspkr                 16384  0
>>>>>>> i2c_piix4              28672  0
>>>>>>> e1000                 163840  0
>>>>>>> i2c_core              106496  2 psmouse,i2c_piix4
>>>>>>> crct10dif_pclmul       16384  1
>>>>>>> crc32_pclmul           16384  0
>>>>>>> ata_generic            16384  0
>>>>>>> pata_acpi              16384  0
>>>>>>> ghash_clmulni_intel    16384  0
>>>>>>> serio_raw              20480  0
>>>>>>> ohci_pci               20480  0
>>>>>>> video                  57344  0
>>>>>>
>>>>>> Yes, this looks more like it.
>>>>>>
>>>>>>> The last 8 entries are the same but a whole load of others are missing.
>>>>>>
>>>>>> Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
>>>>>>
>>>>>> I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
>>>>>>
>>>>>> -Michael
>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> Adolf.
>>>>>>>>> Tried modprobe e1000 but this came back with the following error
>>>>>>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service
>>>>>>>>>
>>>>>>>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Adolf
>>>>>>>>>>> On 11 May 2022, at 20:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi All,
>>>>>>>>>>>
>>>>>>>>>>> On 11/05/2022 20:48, Jon Murphy wrote:
>>>>>>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
>>>>>>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c*
>>>>>>>>>>>> Shouldn’t this be Core Update 168?
>>>>>>>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
>>>>>>>>>>>
>>>>>>>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
>>>>>>>>>>>
>>>>>>>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>
>>>>>>>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>
>>>>>>>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>> That seems to be a bug that should not be there. Did we recently update apache?
>>>>>>>>>> -Michael
>>>>>>>>>>>
>>>>>>>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
>>>>>>>>>>>
>>>>>>>>>>> I think I am going to raise a bug on this.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Adolf.
>>>>>>>>>>>> Jon
>>>>>>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka(a)ipfire.org <mailto:adolf.belka(a)ipfire.org>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote:
>>>>>>>>>>>>>> Hi Leo,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote:
>>>>>>>>>>>>>>> Hi Adolf,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
>>>>>>>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
>>>>>>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 <https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762>
>>>>>>>>>>>>>> Now I see it in the code. Thanks very much.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
>>>>>>>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
>>>>>>>>>>>>>> Will wait to hear other inputs on this problem.
>>>>>>>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
>>>>>>>>>>>>>
>>>>>>>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>> Leo
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> First try:-
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> starting
>>>>>>>>>>>>>>>> DUID 00:04:70:........
>>>>>>>>>>>>>>>> red0: interface not found
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Loading of module with unavailable key is rejected
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Second try:-
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This time running the update took over 10 mins.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Repository was set back to Stable.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Before running the pakfire directory had the following
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> After running it had
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log
>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Rebooted.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Third try:-
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log
>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Rebooted
>>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>>> Several repeats of the following two lines on the console screen:-
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>>>>>>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have given up at this point.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Adolf.
>>>>>>
> 

Re: Feedback on problems with Core Update 168 Testing

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 29302 bytes --]

Please note the patch that I just posted to fix any broken RAID arrays.

Does anyone have any systems that we can use to test this?

-Michael

> On 17 May 2022, at 09:09, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Michael,
> 
> On 13/05/2022 22:37, Michael Tremer wrote:
>> Hello,
>> Okay, this is really bad then if we are breaking RAID installations.
>> I tried to debug this, but VirtualBox is not my friend today.
>> I asked some bug reporter to provide me with the logs that I want over here:
>>   https://bugzilla.ipfire.org/show_bug.cgi?id=12862
> After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
> 
> My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
> 
> Thanks very much for all the help.
> 
> Regards,
> Adolf.
>> -Michael
>>> On 13 May 2022, at 18:12, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>> 
>>> Hi All,
>>> 
>>> I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
>>> 
>>> I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
>>> 
>>> I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
>>> 
>>> The vm shows the following
>>> 
>>> -bash-5.1$ cat /proc/mdstat
>>> Personalities :
>>> md127 : inactive sdb[1](S)
>>>       52428724 blocks super 1.0
>>> 
>>> unused devices: <none>
>>> 
>>> 
>>> It looks like IPFire was only running on sda.
>>> 
>>> So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
>>> 
>>> The CU168 vm that is working has the same as the above setting.
>>> 
>>> The CU168 vm that is not working has the following setting
>>> 
>>> -bash-5.1$ cat /proc/mdstat
>>> Personalities :
>>> md127 : inactive sda[0](S)
>>>       52428724 blocks super 1.0
>>> 
>>> unused devices: <none>
>>> 
>>> So it looks like it has booted up on the disk that was not updated from the raid pair.
>>> 
>>> 
>>> This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
>>> 
>>> It is probably dependent on which of the two disks actually get used to boot from.
>>> 
>>> So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
>>> 
>>> So raid systems need to have their raid arrays fully working before doing an upgrade.
>>> 
>>> At least the above is my interpretation of what I have found. What does everyone else think.
>>> 
>>> 
>>> Regards,
>>> 
>>> Adolf.
>>> 
>>> 
>>> On 13/05/2022 15:59, Adolf Belka wrote:
>>>> Hi Michael,
>>>> 
>>>> I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
>>>> 
>>>> I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
>>>> 
>>>> However I did copy the boot directory info for these two cases and found an interesting effect.
>>>> 
>>>> For the problem vm here was the boot directory layout of the CU167 base machine
>>>> 
>>>> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>>> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
>>>> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
>>>> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>>> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>> 
>>>> After upgrade and before reboot this had changed to
>>>> 
>>>> drwxr-xr-x  5 root root 4.0K May 13 15:40 .
>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>> -rw-r--r--  1 root root 179K May  3 04:02 config-5.15.35-ipfire
>>>> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
>>>> drwxr-xr-x  6 root root 4.0K May 13 15:41 grub
>>>> -rw-------  1 root root  26M May 13 15:41 initramfs-5.15.35-ipfire.img
>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>> -rw-r--r--  1 root root 4.7M May  3 04:02 System.map-5.15.35-ipfire
>>>> -rw-r--r--  1 root root 6.7M May  3 04:02 vmlinuz-5.15.35-ipfire
>>>> 
>>>> Dates and times of some of the files/directories have changed as would be expected.
>>>> 
>>>> After Reboot and with the problem the boot directory was now
>>>> 
>>>> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>>> drwxr-xr-x  3 root root 4.0K Sep 30  2021 efi
>>>> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
>>>> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>>> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>> 
>>>> Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
>>>> 
>>>> For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
>>>> 
>>>> I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
>>>> 
>>>> Regards,
>>>> 
>>>> Adolf.
>>>> 
>>>> On 13/05/2022 12:09, Michael Tremer wrote:
>>>>> Hello,
>>>>> 
>>>>>> On 12 May 2022, at 21:10, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>> 
>>>>>> Hi Michael,
>>>>>> 
>>>>>> On 12/05/2022 14:53, Michael Tremer wrote:
>>>>>>> Hello,
>>>>>>>> On 12 May 2022, at 12:25, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>> 
>>>>>>>> Hi,
>>>>>>>> 
>>>>>>>> On 12/05/2022 11:13, Michael Tremer wrote:
>>>>>>>>> Hello,
>>>>>>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
>>>>>>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
>>>>>>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
>>>>>>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
>>>>>>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
>>>>>>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
>>>>>>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
>>>>>>> Okay. That is indeed quite bad then.
>>>>>>> Since there is no kernel in 168, is there a chance that we broke the update to 167?
>>>>>> I don't know but as far as I have been concerned CU167 has worked well on my vm.
>>>>>> Is there something I should look for on my CU167 vm?
>>>>>>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
>>>>>>> Third time lucky isn’t good enough for me :)
>>>>>> Me neither :-)
>>>>>>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
>>>>>>>> 
>>>>>>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
>>>>>>> Have there been any modules loaded? If one loads, they all should load.
>>>>>> This is what I get with lsmod
>>>>>> 
>>>>>> Module            Size    Used by
>>>>>> crct10dif_pclmul    16384    1
>>>>>> crc32_pclmul        16384    0
>>>>>> ghash_clmulni_intel    16384    0
>>>>>> serio_raw        20480    0
>>>>>> ohci_pci        20480    0
>>>>>> ata_generic        16384    0
>>>>>> pata_acpi        16384    0
>>>>>> video            57344    0
>>>>> 
>>>>> It looks like these modules could have come from the initramdisk image. At least that is working well.
>>>>> 
>>>>>> On my running CU1678 vm lsmod gives the following
>>>>>> 
>>>>>> Module                  Size  Used by
>>>>>> tun                    61440  2
>>>>>> nfnetlink_queue        28672  1
>>>>>> xt_NFQUEUE             16384  8
>>>>>> xt_MASQUERADE          20480  1
>>>>>> cfg80211             1036288  0
>>>>>> rfkill                 32768  1 cfg80211
>>>>>> 8021q                  40960  0
>>>>>> garp                   16384  1 8021q
>>>>>> xt_set                 16384  260
>>>>>> ip_set_hash_net        49152  258
>>>>>> ip_set                 57344  2 xt_set,ip_set_hash_net
>>>>>> xt_hashlimit           20480  2
>>>>>> xt_multiport           20480  4
>>>>>> xt_policy              16384  5
>>>>>> xt_TCPMSS              16384  1
>>>>>> xt_conntrack           16384  7
>>>>>> xt_comment             16384  18
>>>>>> ipt_REJECT             16384  1
>>>>>> nf_reject_ipv4         16384  1 ipt_REJECT
>>>>>> xt_LOG                 20480  26
>>>>>> xt_limit               16384  25
>>>>>> xt_mark                16384  27
>>>>>> xt_connmark            16384  2
>>>>>> nf_log_syslog          24576  26
>>>>>> iptable_raw            16384  0
>>>>>> iptable_mangle         16384  1
>>>>>> iptable_filter         16384  1
>>>>>> vfat                   24576  1
>>>>>> fat                    90112  1 vfat
>>>>>> sch_cake               36864  4
>>>>>> intel_powerclamp       20480  0
>>>>>> psmouse               184320  0
>>>>>> pcspkr                 16384  0
>>>>>> i2c_piix4              28672  0
>>>>>> e1000                 163840  0
>>>>>> i2c_core              106496  2 psmouse,i2c_piix4
>>>>>> crct10dif_pclmul       16384  1
>>>>>> crc32_pclmul           16384  0
>>>>>> ata_generic            16384  0
>>>>>> pata_acpi              16384  0
>>>>>> ghash_clmulni_intel    16384  0
>>>>>> serio_raw              20480  0
>>>>>> ohci_pci               20480  0
>>>>>> video                  57344  0
>>>>> 
>>>>> Yes, this looks more like it.
>>>>> 
>>>>>> The last 8 entries are the same but a whole load of others are missing.
>>>>> 
>>>>> Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
>>>>> 
>>>>> I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
>>>>> 
>>>>> -Michael
>>>>> 
>>>>>> 
>>>>>> Regards,
>>>>>> Adolf.
>>>>>>>> Tried modprobe e1000 but this came back with the following error
>>>>>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service
>>>>>>>> 
>>>>>>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
>>>>>>>> 
>>>>>>>> Regards
>>>>>>>> Adolf
>>>>>>>>>> On 11 May 2022, at 20:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi All,
>>>>>>>>>> 
>>>>>>>>>> On 11/05/2022 20:48, Jon Murphy wrote:
>>>>>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
>>>>>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c*
>>>>>>>>>>> Shouldn’t this be Core Update 168?
>>>>>>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
>>>>>>>>>> 
>>>>>>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
>>>>>>>>>> 
>>>>>>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>> 
>>>>>>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>> 
>>>>>>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>> That seems to be a bug that should not be there. Did we recently update apache?
>>>>>>>>> -Michael
>>>>>>>>>> 
>>>>>>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
>>>>>>>>>> 
>>>>>>>>>> I think I am going to raise a bug on this.
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> Adolf.
>>>>>>>>>>> Jon
>>>>>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka(a)ipfire.org <mailto:adolf.belka(a)ipfire.org>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>> 
>>>>>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote:
>>>>>>>>>>>>> Hi Leo,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote:
>>>>>>>>>>>>>> Hi Adolf,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
>>>>>>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
>>>>>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 <https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762>
>>>>>>>>>>>>> Now I see it in the code. Thanks very much.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
>>>>>>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
>>>>>>>>>>>>> Will wait to hear other inputs on this problem.
>>>>>>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
>>>>>>>>>>>>> 
>>>>>>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
>>>>>>>>>>>> 
>>>>>>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
>>>>>>>>>>>> 
>>>>>>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
>>>>>>>>>>>> 
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> 
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>> 
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>> Leo
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> First try:-
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> starting
>>>>>>>>>>>>>>> DUID 00:04:70:........
>>>>>>>>>>>>>>> red0: interface not found
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Loading of module with unavailable key is rejected
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Second try:-
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> This time running the update took over 10 mins.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Repository was set back to Stable.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Before running the pakfire directory had the following
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> After running it had
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log
>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Rebooted.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Third try:-
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log
>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Rebooted
>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>> Several repeats of the following two lines on the console screen:-
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>>>>>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I have given up at this point.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Adolf.
>>>>> 

Re: Feedback on problems with Core Update 168 Testing

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 31281 bytes --]

Hello Adolf,

thank you for all the extensive testing and reporting back.

Ah, indeed, the newly introduced Cron job is stil missing due to silly me confusing
the fcrontab binary and an actual fcrontab. I think the most straight-forward way to
fix this is to just ship the entire crontab file, and force fcrontab to recreate the
- yes - fcrontab based on its content.

@Michael: How does one ship a single file with a Core Update that is not covered by
any rootfile?

Thanks, and best regards,
Peter Müller


> Hi All,
> 
> Virtually everything I have tested in CU168 looks to be working as expected.
> 
> The only problem I have found to-date is the Rules update for Suricata in the Intrusion Prevention System.
> 
> The old daily or weekly option is gone and the update should now occur twice per day but that entry is not in the fcrontab in CU168 Testing so no updates are occurring for the rulesets unless you manually force them when the update occurs with no problem.
> 
> 
> Regards,
> 
> Adolf
> 
> 
> On 17/05/2022 10:09, Adolf Belka wrote:
>> Hi Michael,
>>
>> On 13/05/2022 22:37, Michael Tremer wrote:
>>> Hello,
>>>
>>> Okay, this is really bad then if we are breaking RAID installations.
>>>
>>> I tried to debug this, but VirtualBox is not my friend today.
>>>
>>> I asked some bug reporter to provide me with the logs that I want over here:
>>>
>>>    https://bugzilla.ipfire.org/show_bug.cgi?id=12862
>> After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
>>
>> My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
>>
>> Thanks very much for all the help.
>>
>> Regards,
>> Adolf.
>>>
>>> -Michael
>>>
>>>> On 13 May 2022, at 18:12, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>
>>>> Hi All,
>>>>
>>>> I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
>>>>
>>>> I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
>>>>
>>>> I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
>>>>
>>>> The vm shows the following
>>>>
>>>> -bash-5.1$ cat /proc/mdstat
>>>> Personalities :
>>>> md127 : inactive sdb[1](S)
>>>>        52428724 blocks super 1.0
>>>>
>>>> unused devices: <none>
>>>>
>>>>
>>>> It looks like IPFire was only running on sda.
>>>>
>>>> So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
>>>>
>>>> The CU168 vm that is working has the same as the above setting.
>>>>
>>>> The CU168 vm that is not working has the following setting
>>>>
>>>> -bash-5.1$ cat /proc/mdstat
>>>> Personalities :
>>>> md127 : inactive sda[0](S)
>>>>        52428724 blocks super 1.0
>>>>
>>>> unused devices: <none>
>>>>
>>>> So it looks like it has booted up on the disk that was not updated from the raid pair.
>>>>
>>>>
>>>> This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
>>>>
>>>> It is probably dependent on which of the two disks actually get used to boot from.
>>>>
>>>> So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
>>>>
>>>> So raid systems need to have their raid arrays fully working before doing an upgrade.
>>>>
>>>> At least the above is my interpretation of what I have found. What does everyone else think.
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Adolf.
>>>>
>>>>
>>>> On 13/05/2022 15:59, Adolf Belka wrote:
>>>>> Hi Michael,
>>>>>
>>>>> I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
>>>>>
>>>>> I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
>>>>>
>>>>> However I did copy the boot directory info for these two cases and found an interesting effect.
>>>>>
>>>>> For the problem vm here was the boot directory layout of the CU167 base machine
>>>>>
>>>>> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>>>> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
>>>>> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
>>>>> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>>> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>>>> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>>>
>>>>> After upgrade and before reboot this had changed to
>>>>>
>>>>> drwxr-xr-x  5 root root 4.0K May 13 15:40 .
>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>> -rw-r--r--  1 root root 179K May  3 04:02 config-5.15.35-ipfire
>>>>> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
>>>>> drwxr-xr-x  6 root root 4.0K May 13 15:41 grub
>>>>> -rw-------  1 root root  26M May 13 15:41 initramfs-5.15.35-ipfire.img
>>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>>> -rw-r--r--  1 root root 4.7M May  3 04:02 System.map-5.15.35-ipfire
>>>>> -rw-r--r--  1 root root 6.7M May  3 04:02 vmlinuz-5.15.35-ipfire
>>>>>
>>>>> Dates and times of some of the files/directories have changed as would be expected.
>>>>>
>>>>> After Reboot and with the problem the boot directory was now
>>>>>
>>>>> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
>>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>>> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>>>> drwxr-xr-x  3 root root 4.0K Sep 30  2021 efi
>>>>> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
>>>>> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>>> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>>>> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>>>
>>>>> Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
>>>>>
>>>>> For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
>>>>>
>>>>> I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Adolf.
>>>>>
>>>>> On 13/05/2022 12:09, Michael Tremer wrote:
>>>>>> Hello,
>>>>>>
>>>>>>> On 12 May 2022, at 21:10, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>
>>>>>>> Hi Michael,
>>>>>>>
>>>>>>> On 12/05/2022 14:53, Michael Tremer wrote:
>>>>>>>> Hello,
>>>>>>>>> On 12 May 2022, at 12:25, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On 12/05/2022 11:13, Michael Tremer wrote:
>>>>>>>>>> Hello,
>>>>>>>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
>>>>>>>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
>>>>>>>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
>>>>>>>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
>>>>>>>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
>>>>>>>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
>>>>>>>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
>>>>>>>> Okay. That is indeed quite bad then.
>>>>>>>> Since there is no kernel in 168, is there a chance that we broke the update to 167?
>>>>>>> I don't know but as far as I have been concerned CU167 has worked well on my vm.
>>>>>>> Is there something I should look for on my CU167 vm?
>>>>>>>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
>>>>>>>> Third time lucky isn’t good enough for me :)
>>>>>>> Me neither :-)
>>>>>>>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
>>>>>>>>>
>>>>>>>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
>>>>>>>> Have there been any modules loaded? If one loads, they all should load.
>>>>>>> This is what I get with lsmod
>>>>>>>
>>>>>>> Module            Size    Used by
>>>>>>> crct10dif_pclmul    16384    1
>>>>>>> crc32_pclmul        16384    0
>>>>>>> ghash_clmulni_intel    16384    0
>>>>>>> serio_raw        20480    0
>>>>>>> ohci_pci        20480    0
>>>>>>> ata_generic        16384    0
>>>>>>> pata_acpi        16384    0
>>>>>>> video            57344    0
>>>>>>
>>>>>> It looks like these modules could have come from the initramdisk image. At least that is working well.
>>>>>>
>>>>>>> On my running CU1678 vm lsmod gives the following
>>>>>>>
>>>>>>> Module                  Size  Used by
>>>>>>> tun                    61440  2
>>>>>>> nfnetlink_queue        28672  1
>>>>>>> xt_NFQUEUE             16384  8
>>>>>>> xt_MASQUERADE          20480  1
>>>>>>> cfg80211             1036288  0
>>>>>>> rfkill                 32768  1 cfg80211
>>>>>>> 8021q                  40960  0
>>>>>>> garp                   16384  1 8021q
>>>>>>> xt_set                 16384  260
>>>>>>> ip_set_hash_net        49152  258
>>>>>>> ip_set                 57344  2 xt_set,ip_set_hash_net
>>>>>>> xt_hashlimit           20480  2
>>>>>>> xt_multiport           20480  4
>>>>>>> xt_policy              16384  5
>>>>>>> xt_TCPMSS              16384  1
>>>>>>> xt_conntrack           16384  7
>>>>>>> xt_comment             16384  18
>>>>>>> ipt_REJECT             16384  1
>>>>>>> nf_reject_ipv4         16384  1 ipt_REJECT
>>>>>>> xt_LOG                 20480  26
>>>>>>> xt_limit               16384  25
>>>>>>> xt_mark                16384  27
>>>>>>> xt_connmark            16384  2
>>>>>>> nf_log_syslog          24576  26
>>>>>>> iptable_raw            16384  0
>>>>>>> iptable_mangle         16384  1
>>>>>>> iptable_filter         16384  1
>>>>>>> vfat                   24576  1
>>>>>>> fat                    90112  1 vfat
>>>>>>> sch_cake               36864  4
>>>>>>> intel_powerclamp       20480  0
>>>>>>> psmouse               184320  0
>>>>>>> pcspkr                 16384  0
>>>>>>> i2c_piix4              28672  0
>>>>>>> e1000                 163840  0
>>>>>>> i2c_core              106496  2 psmouse,i2c_piix4
>>>>>>> crct10dif_pclmul       16384  1
>>>>>>> crc32_pclmul           16384  0
>>>>>>> ata_generic            16384  0
>>>>>>> pata_acpi              16384  0
>>>>>>> ghash_clmulni_intel    16384  0
>>>>>>> serio_raw              20480  0
>>>>>>> ohci_pci               20480  0
>>>>>>> video                  57344  0
>>>>>>
>>>>>> Yes, this looks more like it.
>>>>>>
>>>>>>> The last 8 entries are the same but a whole load of others are missing.
>>>>>>
>>>>>> Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
>>>>>>
>>>>>> I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
>>>>>>
>>>>>> -Michael
>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> Adolf.
>>>>>>>>> Tried modprobe e1000 but this came back with the following error
>>>>>>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service
>>>>>>>>>
>>>>>>>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Adolf
>>>>>>>>>>> On 11 May 2022, at 20:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi All,
>>>>>>>>>>>
>>>>>>>>>>> On 11/05/2022 20:48, Jon Murphy wrote:
>>>>>>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
>>>>>>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c*
>>>>>>>>>>>> Shouldn’t this be Core Update 168?
>>>>>>>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
>>>>>>>>>>>
>>>>>>>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
>>>>>>>>>>>
>>>>>>>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>
>>>>>>>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>>
>>>>>>>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>> That seems to be a bug that should not be there. Did we recently update apache?
>>>>>>>>>> -Michael
>>>>>>>>>>>
>>>>>>>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
>>>>>>>>>>>
>>>>>>>>>>> I think I am going to raise a bug on this.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Adolf.
>>>>>>>>>>>> Jon
>>>>>>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka(a)ipfire.org <mailto:adolf.belka(a)ipfire.org>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote:
>>>>>>>>>>>>>> Hi Leo,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote:
>>>>>>>>>>>>>>> Hi Adolf,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
>>>>>>>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
>>>>>>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 <https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762>
>>>>>>>>>>>>>> Now I see it in the code. Thanks very much.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
>>>>>>>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
>>>>>>>>>>>>>> Will wait to hear other inputs on this problem.
>>>>>>>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
>>>>>>>>>>>>>
>>>>>>>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>> Leo
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> First try:-
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> starting
>>>>>>>>>>>>>>>> DUID 00:04:70:........
>>>>>>>>>>>>>>>> red0: interface not found
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Loading of module with unavailable key is rejected
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Second try:-
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This time running the update took over 10 mins.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Repository was set back to Stable.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Before running the pakfire directory had the following
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> After running it had
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log
>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Rebooted.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Third try:-
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log
>>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Rebooted
>>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>>> Several repeats of the following two lines on the console screen:-
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>>>>>>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have given up at this point.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Adolf.
>>>>>>
>>>

Re: Feedback on problems with Core Update 168 Testing

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 30241 bytes --]

Hi All,

Virtually everything I have tested in CU168 looks to be working as expected.

The only problem I have found to-date is the Rules update for Suricata in the Intrusion Prevention System.

The old daily or weekly option is gone and the update should now occur twice per day but that entry is not in the fcrontab in CU168 Testing so no updates are occurring for the rulesets unless you manually force them when the update occurs with no problem.


Regards,

Adolf


On 17/05/2022 10:09, Adolf Belka wrote:
> Hi Michael,
>
> On 13/05/2022 22:37, Michael Tremer wrote:
>> Hello,
>>
>> Okay, this is really bad then if we are breaking RAID installations.
>>
>> I tried to debug this, but VirtualBox is not my friend today.
>>
>> I asked some bug reporter to provide me with the logs that I want over here:
>>
>>    https://bugzilla.ipfire.org/show_bug.cgi?id=12862
> After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.
>
> My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.
>
> Thanks very much for all the help.
>
> Regards,
> Adolf.
>>
>> -Michael
>>
>>> On 13 May 2022, at 18:12, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>
>>> Hi All,
>>>
>>> I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
>>>
>>> I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
>>>
>>> I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
>>>
>>> The vm shows the following
>>>
>>> -bash-5.1$ cat /proc/mdstat
>>> Personalities :
>>> md127 : inactive sdb[1](S)
>>>        52428724 blocks super 1.0
>>>
>>> unused devices: <none>
>>>
>>>
>>> It looks like IPFire was only running on sda.
>>>
>>> So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
>>>
>>> The CU168 vm that is working has the same as the above setting.
>>>
>>> The CU168 vm that is not working has the following setting
>>>
>>> -bash-5.1$ cat /proc/mdstat
>>> Personalities :
>>> md127 : inactive sda[0](S)
>>>        52428724 blocks super 1.0
>>>
>>> unused devices: <none>
>>>
>>> So it looks like it has booted up on the disk that was not updated from the raid pair.
>>>
>>>
>>> This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
>>>
>>> It is probably dependent on which of the two disks actually get used to boot from.
>>>
>>> So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
>>>
>>> So raid systems need to have their raid arrays fully working before doing an upgrade.
>>>
>>> At least the above is my interpretation of what I have found. What does everyone else think.
>>>
>>>
>>> Regards,
>>>
>>> Adolf.
>>>
>>>
>>> On 13/05/2022 15:59, Adolf Belka wrote:
>>>> Hi Michael,
>>>>
>>>> I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
>>>>
>>>> I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
>>>>
>>>> However I did copy the boot directory info for these two cases and found an interesting effect.
>>>>
>>>> For the problem vm here was the boot directory layout of the CU167 base machine
>>>>
>>>> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>>> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
>>>> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
>>>> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>>> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>>
>>>> After upgrade and before reboot this had changed to
>>>>
>>>> drwxr-xr-x  5 root root 4.0K May 13 15:40 .
>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>> -rw-r--r--  1 root root 179K May  3 04:02 config-5.15.35-ipfire
>>>> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
>>>> drwxr-xr-x  6 root root 4.0K May 13 15:41 grub
>>>> -rw-------  1 root root  26M May 13 15:41 initramfs-5.15.35-ipfire.img
>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>> -rw-r--r--  1 root root 4.7M May  3 04:02 System.map-5.15.35-ipfire
>>>> -rw-r--r--  1 root root 6.7M May  3 04:02 vmlinuz-5.15.35-ipfire
>>>>
>>>> Dates and times of some of the files/directories have changed as would be expected.
>>>>
>>>> After Reboot and with the problem the boot directory was now
>>>>
>>>> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
>>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>>> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>>> drwxr-xr-x  3 root root 4.0K Sep 30  2021 efi
>>>> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
>>>> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>>> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>>> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>>
>>>> Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
>>>>
>>>> For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
>>>>
>>>> I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
>>>>
>>>> Regards,
>>>>
>>>> Adolf.
>>>>
>>>> On 13/05/2022 12:09, Michael Tremer wrote:
>>>>> Hello,
>>>>>
>>>>>> On 12 May 2022, at 21:10, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>
>>>>>> Hi Michael,
>>>>>>
>>>>>> On 12/05/2022 14:53, Michael Tremer wrote:
>>>>>>> Hello,
>>>>>>>> On 12 May 2022, at 12:25, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On 12/05/2022 11:13, Michael Tremer wrote:
>>>>>>>>> Hello,
>>>>>>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
>>>>>>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
>>>>>>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
>>>>>>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
>>>>>>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
>>>>>>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
>>>>>>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
>>>>>>> Okay. That is indeed quite bad then.
>>>>>>> Since there is no kernel in 168, is there a chance that we broke the update to 167?
>>>>>> I don't know but as far as I have been concerned CU167 has worked well on my vm.
>>>>>> Is there something I should look for on my CU167 vm?
>>>>>>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
>>>>>>> Third time lucky isn’t good enough for me :)
>>>>>> Me neither :-)
>>>>>>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
>>>>>>>>
>>>>>>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
>>>>>>> Have there been any modules loaded? If one loads, they all should load.
>>>>>> This is what I get with lsmod
>>>>>>
>>>>>> Module            Size    Used by
>>>>>> crct10dif_pclmul    16384    1
>>>>>> crc32_pclmul        16384    0
>>>>>> ghash_clmulni_intel    16384    0
>>>>>> serio_raw        20480    0
>>>>>> ohci_pci        20480    0
>>>>>> ata_generic        16384    0
>>>>>> pata_acpi        16384    0
>>>>>> video            57344    0
>>>>>
>>>>> It looks like these modules could have come from the initramdisk image. At least that is working well.
>>>>>
>>>>>> On my running CU1678 vm lsmod gives the following
>>>>>>
>>>>>> Module                  Size  Used by
>>>>>> tun                    61440  2
>>>>>> nfnetlink_queue        28672  1
>>>>>> xt_NFQUEUE             16384  8
>>>>>> xt_MASQUERADE          20480  1
>>>>>> cfg80211             1036288  0
>>>>>> rfkill                 32768  1 cfg80211
>>>>>> 8021q                  40960  0
>>>>>> garp                   16384  1 8021q
>>>>>> xt_set                 16384  260
>>>>>> ip_set_hash_net        49152  258
>>>>>> ip_set                 57344  2 xt_set,ip_set_hash_net
>>>>>> xt_hashlimit           20480  2
>>>>>> xt_multiport           20480  4
>>>>>> xt_policy              16384  5
>>>>>> xt_TCPMSS              16384  1
>>>>>> xt_conntrack           16384  7
>>>>>> xt_comment             16384  18
>>>>>> ipt_REJECT             16384  1
>>>>>> nf_reject_ipv4         16384  1 ipt_REJECT
>>>>>> xt_LOG                 20480  26
>>>>>> xt_limit               16384  25
>>>>>> xt_mark                16384  27
>>>>>> xt_connmark            16384  2
>>>>>> nf_log_syslog          24576  26
>>>>>> iptable_raw            16384  0
>>>>>> iptable_mangle         16384  1
>>>>>> iptable_filter         16384  1
>>>>>> vfat                   24576  1
>>>>>> fat                    90112  1 vfat
>>>>>> sch_cake               36864  4
>>>>>> intel_powerclamp       20480  0
>>>>>> psmouse               184320  0
>>>>>> pcspkr                 16384  0
>>>>>> i2c_piix4              28672  0
>>>>>> e1000                 163840  0
>>>>>> i2c_core              106496  2 psmouse,i2c_piix4
>>>>>> crct10dif_pclmul       16384  1
>>>>>> crc32_pclmul           16384  0
>>>>>> ata_generic            16384  0
>>>>>> pata_acpi              16384  0
>>>>>> ghash_clmulni_intel    16384  0
>>>>>> serio_raw              20480  0
>>>>>> ohci_pci               20480  0
>>>>>> video                  57344  0
>>>>>
>>>>> Yes, this looks more like it.
>>>>>
>>>>>> The last 8 entries are the same but a whole load of others are missing.
>>>>>
>>>>> Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
>>>>>
>>>>> I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
>>>>>
>>>>> -Michael
>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Adolf.
>>>>>>>> Tried modprobe e1000 but this came back with the following error
>>>>>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service
>>>>>>>>
>>>>>>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Adolf
>>>>>>>>>> On 11 May 2022, at 20:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi All,
>>>>>>>>>>
>>>>>>>>>> On 11/05/2022 20:48, Jon Murphy wrote:
>>>>>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
>>>>>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c*
>>>>>>>>>>> Shouldn’t this be Core Update 168?
>>>>>>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
>>>>>>>>>>
>>>>>>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
>>>>>>>>>>
>>>>>>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>
>>>>>>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>>
>>>>>>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>> That seems to be a bug that should not be there. Did we recently update apache?
>>>>>>>>> -Michael
>>>>>>>>>>
>>>>>>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
>>>>>>>>>>
>>>>>>>>>> I think I am going to raise a bug on this.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Adolf.
>>>>>>>>>>> Jon
>>>>>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka(a)ipfire.org <mailto:adolf.belka(a)ipfire.org>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>
>>>>>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote:
>>>>>>>>>>>>> Hi Leo,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote:
>>>>>>>>>>>>>> Hi Adolf,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
>>>>>>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
>>>>>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 <https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762>
>>>>>>>>>>>>> Now I see it in the code. Thanks very much.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
>>>>>>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
>>>>>>>>>>>>> Will wait to hear other inputs on this problem.
>>>>>>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
>>>>>>>>>>>>>
>>>>>>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
>>>>>>>>>>>>
>>>>>>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
>>>>>>>>>>>>
>>>>>>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>> Leo
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> First try:-
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> starting
>>>>>>>>>>>>>>> DUID 00:04:70:........
>>>>>>>>>>>>>>> red0: interface not found
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Loading of module with unavailable key is rejected
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Second try:-
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This time running the update took over 10 mins.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Repository was set back to Stable.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Before running the pakfire directory had the following
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> After running it had
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log
>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Rebooted.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Third try:-
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log
>>>>>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Rebooted
>>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>> Several repeats of the following two lines on the console screen:-
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>>>>>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have given up at this point.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Adolf.
>>>>>
>>

Re: Feedback on problems with Core Update 168 Testing

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 28477 bytes --]

Hi Michael,

On 13/05/2022 22:37, Michael Tremer wrote:
> Hello,
> 
> Okay, this is really bad then if we are breaking RAID installations.
> 
> I tried to debug this, but VirtualBox is not my friend today.
> 
> I asked some bug reporter to provide me with the logs that I want over here:
> 
>    https://bugzilla.ipfire.org/show_bug.cgi?id=12862
After applying this fix to my CU167 vm and proving that it stayed consistent I have now done an upgrade to CU168 Testing. This does a CU167 update before doing the CU168 update and I am glad to announce that my previous problems with losing all my network interfaces etc are gone.

My Core Update 168 Development Build: master/bbd4767f successfully restarted with a fully active raid array and I will now restart doing my CU168 Testing evaluation.

Thanks very much for all the help.

Regards,
Adolf.
> 
> -Michael
> 
>> On 13 May 2022, at 18:12, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi All,
>>
>> I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
>>
>> I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
>>
>> I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
>>
>> The vm shows the following
>>
>> -bash-5.1$ cat /proc/mdstat
>> Personalities :
>> md127 : inactive sdb[1](S)
>>        52428724 blocks super 1.0
>>
>> unused devices: <none>
>>
>>
>> It looks like IPFire was only running on sda.
>>
>> So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
>>
>> The CU168 vm that is working has the same as the above setting.
>>
>> The CU168 vm that is not working has the following setting
>>
>> -bash-5.1$ cat /proc/mdstat
>> Personalities :
>> md127 : inactive sda[0](S)
>>        52428724 blocks super 1.0
>>
>> unused devices: <none>
>>
>> So it looks like it has booted up on the disk that was not updated from the raid pair.
>>
>>
>> This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
>>
>> It is probably dependent on which of the two disks actually get used to boot from.
>>
>> So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
>>
>> So raid systems need to have their raid arrays fully working before doing an upgrade.
>>
>> At least the above is my interpretation of what I have found. What does everyone else think.
>>
>>
>> Regards,
>>
>> Adolf.
>>
>>
>> On 13/05/2022 15:59, Adolf Belka wrote:
>>> Hi Michael,
>>>
>>> I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
>>>
>>> I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
>>>
>>> However I did copy the boot directory info for these two cases and found an interesting effect.
>>>
>>> For the problem vm here was the boot directory layout of the CU167 base machine
>>>
>>> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
>>> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
>>> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>
>>> After upgrade and before reboot this had changed to
>>>
>>> drwxr-xr-x  5 root root 4.0K May 13 15:40 .
>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>> -rw-r--r--  1 root root 179K May  3 04:02 config-5.15.35-ipfire
>>> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
>>> drwxr-xr-x  6 root root 4.0K May 13 15:41 grub
>>> -rw-------  1 root root  26M May 13 15:41 initramfs-5.15.35-ipfire.img
>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>> -rw-r--r--  1 root root 4.7M May  3 04:02 System.map-5.15.35-ipfire
>>> -rw-r--r--  1 root root 6.7M May  3 04:02 vmlinuz-5.15.35-ipfire
>>>
>>> Dates and times of some of the files/directories have changed as would be expected.
>>>
>>> After Reboot and with the problem the boot directory was now
>>>
>>> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
>>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>>> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>>> drwxr-xr-x  3 root root 4.0K Sep 30  2021 efi
>>> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
>>> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>>> drwx------  2 root root  16K Sep 30  2021 lost+found
>>> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>>> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>>>
>>> Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
>>>
>>> For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
>>>
>>> I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
>>>
>>> Regards,
>>>
>>> Adolf.
>>>
>>> On 13/05/2022 12:09, Michael Tremer wrote:
>>>> Hello,
>>>>
>>>>> On 12 May 2022, at 21:10, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>
>>>>> Hi Michael,
>>>>>
>>>>> On 12/05/2022 14:53, Michael Tremer wrote:
>>>>>> Hello,
>>>>>>> On 12 May 2022, at 12:25, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> On 12/05/2022 11:13, Michael Tremer wrote:
>>>>>>>> Hello,
>>>>>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
>>>>>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
>>>>>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
>>>>>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
>>>>>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
>>>>>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
>>>>>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
>>>>>> Okay. That is indeed quite bad then.
>>>>>> Since there is no kernel in 168, is there a chance that we broke the update to 167?
>>>>> I don't know but as far as I have been concerned CU167 has worked well on my vm.
>>>>> Is there something I should look for on my CU167 vm?
>>>>>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
>>>>>> Third time lucky isn’t good enough for me :)
>>>>> Me neither :-)
>>>>>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
>>>>>>>
>>>>>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
>>>>>> Have there been any modules loaded? If one loads, they all should load.
>>>>> This is what I get with lsmod
>>>>>
>>>>> Module            Size    Used by
>>>>> crct10dif_pclmul    16384    1
>>>>> crc32_pclmul        16384    0
>>>>> ghash_clmulni_intel    16384    0
>>>>> serio_raw        20480    0
>>>>> ohci_pci        20480    0
>>>>> ata_generic        16384    0
>>>>> pata_acpi        16384    0
>>>>> video            57344    0
>>>>
>>>> It looks like these modules could have come from the initramdisk image. At least that is working well.
>>>>
>>>>> On my running CU1678 vm lsmod gives the following
>>>>>
>>>>> Module                  Size  Used by
>>>>> tun                    61440  2
>>>>> nfnetlink_queue        28672  1
>>>>> xt_NFQUEUE             16384  8
>>>>> xt_MASQUERADE          20480  1
>>>>> cfg80211             1036288  0
>>>>> rfkill                 32768  1 cfg80211
>>>>> 8021q                  40960  0
>>>>> garp                   16384  1 8021q
>>>>> xt_set                 16384  260
>>>>> ip_set_hash_net        49152  258
>>>>> ip_set                 57344  2 xt_set,ip_set_hash_net
>>>>> xt_hashlimit           20480  2
>>>>> xt_multiport           20480  4
>>>>> xt_policy              16384  5
>>>>> xt_TCPMSS              16384  1
>>>>> xt_conntrack           16384  7
>>>>> xt_comment             16384  18
>>>>> ipt_REJECT             16384  1
>>>>> nf_reject_ipv4         16384  1 ipt_REJECT
>>>>> xt_LOG                 20480  26
>>>>> xt_limit               16384  25
>>>>> xt_mark                16384  27
>>>>> xt_connmark            16384  2
>>>>> nf_log_syslog          24576  26
>>>>> iptable_raw            16384  0
>>>>> iptable_mangle         16384  1
>>>>> iptable_filter         16384  1
>>>>> vfat                   24576  1
>>>>> fat                    90112  1 vfat
>>>>> sch_cake               36864  4
>>>>> intel_powerclamp       20480  0
>>>>> psmouse               184320  0
>>>>> pcspkr                 16384  0
>>>>> i2c_piix4              28672  0
>>>>> e1000                 163840  0
>>>>> i2c_core              106496  2 psmouse,i2c_piix4
>>>>> crct10dif_pclmul       16384  1
>>>>> crc32_pclmul           16384  0
>>>>> ata_generic            16384  0
>>>>> pata_acpi              16384  0
>>>>> ghash_clmulni_intel    16384  0
>>>>> serio_raw              20480  0
>>>>> ohci_pci               20480  0
>>>>> video                  57344  0
>>>>
>>>> Yes, this looks more like it.
>>>>
>>>>> The last 8 entries are the same but a whole load of others are missing.
>>>>
>>>> Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
>>>>
>>>> I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
>>>>
>>>> -Michael
>>>>
>>>>>
>>>>> Regards,
>>>>> Adolf.
>>>>>>> Tried modprobe e1000 but this came back with the following error
>>>>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service
>>>>>>>
>>>>>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
>>>>>>>
>>>>>>> Regards
>>>>>>> Adolf
>>>>>>>>> On 11 May 2022, at 20:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>>>
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> On 11/05/2022 20:48, Jon Murphy wrote:
>>>>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
>>>>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c*
>>>>>>>>>> Shouldn’t this be Core Update 168?
>>>>>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
>>>>>>>>>
>>>>>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
>>>>>>>>>
>>>>>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>
>>>>>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
>>>>>>>>>
>>>>>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
>>>>>>>> That seems to be a bug that should not be there. Did we recently update apache?
>>>>>>>> -Michael
>>>>>>>>>
>>>>>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
>>>>>>>>>
>>>>>>>>> I think I am going to raise a bug on this.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Adolf.
>>>>>>>>>> Jon
>>>>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka(a)ipfire.org <mailto:adolf.belka(a)ipfire.org>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi All,
>>>>>>>>>>>
>>>>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote:
>>>>>>>>>>>> Hi Leo,
>>>>>>>>>>>>
>>>>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote:
>>>>>>>>>>>>> Hi Adolf,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
>>>>>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
>>>>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 <https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762>
>>>>>>>>>>>> Now I see it in the code. Thanks very much.
>>>>>>>>>>>>>
>>>>>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
>>>>>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions.
>>>>>>>>>>>>
>>>>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
>>>>>>>>>>>> Will wait to hear other inputs on this problem.
>>>>>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
>>>>>>>>>>>>
>>>>>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
>>>>>>>>>>>
>>>>>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
>>>>>>>>>>>
>>>>>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>> Adolf.
>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Adolf.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>> Leo
>>>>>>>>>>>>>
>>>>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> First try:-
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> starting
>>>>>>>>>>>>>> DUID 00:04:70:........
>>>>>>>>>>>>>> red0: interface not found
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Loading of module with unavailable key is rejected
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Second try:-
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This time running the update took over 10 mins.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Repository was set back to Stable.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Before running the pakfire directory had the following
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> After running it had
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log
>>>>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Rebooted.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Third try:-
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log
>>>>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Rebooted
>>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>>> Several repeats of the following two lines on the console screen:-
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>>>>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have given up at this point.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Adolf.
>>>>
> 

Re: Feedback on problems with Core Update 168 Testing

[Rob Brewer]

[-- Attachment #1: Type: text/plain, Size: 2525 bytes --]

On Friday 13 May 2022 22:18 Rob Brewer wrote:

> On Friday 13 May 2022 21:36 Michael Tremer wrote:
> 
>> Hello,
>> 
>>> On 13 May 2022, at 16:02, Rob Brewer <ipfire-devel(a)grantura.co.uk>
>>> wrote:
>>>
>>> On Thursday 12 May 2022 14:15 Rob Brewer wrote:
>>>
>>>> On Thursday 12 May 2022 13:51 Michael Tremer wrote:
>>>>
>>>>> Hello Rob,
>>>>>
>>>>> You seem to already have lost the kernel there which is not part of
>>>>> 168.
>>>>>
>>>>> Can you extract any log files from /var/log/pakfire?
>>>>>
>>>>> -Michael
>>>>>
>>>> I was upgrading 165-168 so I think Linux 5.15.23-ipfire was from 165.
>>>> I am trying to build a bootable USB drive so I can mount the ssd and
>>>> look at the Pakfire logs.
>>>>
>>>>
>>>> Rob
>>>>
>>> I have been unable to boot from a usb drive so I re-fitted an old mSATA
>>> SSD with an earlier version of IPFIRE and was surprisingly unable to
>>> boot from there also.
>>>
>>> On investigation the bios is reporting:
>>>
>>> SeaBIOS (version rel-1.14.0.1-0-g8610266a)
>>>
>>> yet I manually upgraded it to v4.16.0.2 a few weeks ago after a
>>> discussion with Bernard.
>>>
>>> https://community.ipfire.org/t/entropy-with-new-apu-firmware/7707
>>>
>>> Could something have downgraded or corrupted the BIOS?
>> 
>> No, we never touch this as it is too unpredictable what would happen.
>> 
>>> I'll try to re-flash back to v4.16.0.2 if I can and try to boot again.
>> 
>> But that is not the same version number that you are comparing there.
>> 
>> SeaBIOS is a part of the firmware and comes in a different version.
> 
> Yes sorry the BIOS version was a red-herring. When I eventually could boot
> from USB I could see that the correct BIOS version was displayed on the
> boot screen.
> 
> My problem seems to be that grub hasn't been correctly updated to load the
> current kernel.
> 

On checking the pakfire log files it would seem that grub wasn't updated 
after  kernel 5.15.35 was introduced in core 167. 

I suspect this was caused by a failure in 'dracut'  :

tail update-core-upgrade-167.log 

dracut: Executing: /usr/bin/dracut --kver=5.15.35-ipfire --force
dracut: *** Including module: modsign ***
dracut: *** Including module: i18n ***
dracut: *** Including module: dm ***
dracut: Skipping udev rule: 64-device-mapper.rules
dracut: Skipping udev rule: 60-persistent-storage-dm.rules
dracut: Skipping udev rule: 55-dm.rules
dracut: *** Including module: kernel-modules ***

I presume this should have updated grub but dracut is new to me so I'm not 
sure what to expect here!


Rob

Re: Feedback on problems with Core Update 168 Testing

[Rob Brewer]

[-- Attachment #1: Type: text/plain, Size: 1707 bytes --]

On Friday 13 May 2022 21:36 Michael Tremer wrote:

> Hello,
> 
>> On 13 May 2022, at 16:02, Rob Brewer <ipfire-devel(a)grantura.co.uk> wrote:
>>
>> On Thursday 12 May 2022 14:15 Rob Brewer wrote:
>>
>>> On Thursday 12 May 2022 13:51 Michael Tremer wrote:
>>>
>>>> Hello Rob,
>>>>
>>>> You seem to already have lost the kernel there which is not part of
>>>> 168.
>>>>
>>>> Can you extract any log files from /var/log/pakfire?
>>>>
>>>> -Michael
>>>>
>>> I was upgrading 165-168 so I think Linux 5.15.23-ipfire was from 165.
>>> I am trying to build a bootable USB drive so I can mount the ssd and
>>> look at the Pakfire logs.
>>>
>>>
>>> Rob
>>>
>> I have been unable to boot from a usb drive so I re-fitted an old mSATA
>> SSD with an earlier version of IPFIRE and was surprisingly unable to boot
>> from there also.
>>
>> On investigation the bios is reporting:
>>
>> SeaBIOS (version rel-1.14.0.1-0-g8610266a)
>>
>> yet I manually upgraded it to v4.16.0.2 a few weeks ago after a
>> discussion with Bernard.
>>
>> https://community.ipfire.org/t/entropy-with-new-apu-firmware/7707
>>
>> Could something have downgraded or corrupted the BIOS?
> 
> No, we never touch this as it is too unpredictable what would happen.
> 
>> I'll try to re-flash back to v4.16.0.2 if I can and try to boot again.
> 
> But that is not the same version number that you are comparing there.
> 
> SeaBIOS is a part of the firmware and comes in a different version.

Yes sorry the BIOS version was a red-herring. When I eventually could boot 
from USB I could see that the correct BIOS version was displayed on the boot 
screen.

My problem seems to be that grub hasn't been correctly updated to load the 
current kernel. 

Rob

Re: Feedback on problems with Core Update 168 Testing

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 27558 bytes --]

Hello,

Okay, this is really bad then if we are breaking RAID installations.

I tried to debug this, but VirtualBox is not my friend today.

I asked some bug reporter to provide me with the logs that I want over here:

  https://bugzilla.ipfire.org/show_bug.cgi?id=12862

-Michael

> On 13 May 2022, at 18:12, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi All,
> 
> I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.
> 
> I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.
> 
> I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.
> 
> The vm shows the following
> 
> -bash-5.1$ cat /proc/mdstat
> Personalities :
> md127 : inactive sdb[1](S)
>       52428724 blocks super 1.0
> 
> unused devices: <none>
> 
> 
> It looks like IPFire was only running on sda.
> 
> So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.
> 
> The CU168 vm that is working has the same as the above setting.
> 
> The CU168 vm that is not working has the following setting
> 
> -bash-5.1$ cat /proc/mdstat
> Personalities :
> md127 : inactive sda[0](S)
>       52428724 blocks super 1.0
> 
> unused devices: <none>
> 
> So it looks like it has booted up on the disk that was not updated from the raid pair.
> 
> 
> This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.
> 
> It is probably dependent on which of the two disks actually get used to boot from.
> 
> So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.
> 
> So raid systems need to have their raid arrays fully working before doing an upgrade.
> 
> At least the above is my interpretation of what I have found. What does everyone else think.
> 
> 
> Regards,
> 
> Adolf.
> 
> 
> On 13/05/2022 15:59, Adolf Belka wrote:
>> Hi Michael,
>> 
>> I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
>> 
>> I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
>> 
>> However I did copy the boot directory info for these two cases and found an interesting effect.
>> 
>> For the problem vm here was the boot directory layout of the CU167 base machine
>> 
>> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
>> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
>> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>> drwx------  2 root root  16K Sep 30  2021 lost+found
>> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>> 
>> After upgrade and before reboot this had changed to
>> 
>> drwxr-xr-x  5 root root 4.0K May 13 15:40 .
>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>> -rw-r--r--  1 root root 179K May  3 04:02 config-5.15.35-ipfire
>> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
>> drwxr-xr-x  6 root root 4.0K May 13 15:41 grub
>> -rw-------  1 root root  26M May 13 15:41 initramfs-5.15.35-ipfire.img
>> drwx------  2 root root  16K Sep 30  2021 lost+found
>> -rw-r--r--  1 root root 4.7M May  3 04:02 System.map-5.15.35-ipfire
>> -rw-r--r--  1 root root 6.7M May  3 04:02 vmlinuz-5.15.35-ipfire
>> 
>> Dates and times of some of the files/directories have changed as would be expected.
>> 
>> After Reboot and with the problem the boot directory was now
>> 
>> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
>> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
>> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
>> drwxr-xr-x  3 root root 4.0K Sep 30  2021 efi
>> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
>> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
>> drwx------  2 root root  16K Sep 30  2021 lost+found
>> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
>> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>> 
>> Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
>> 
>> For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
>> 
>> I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
>> 
>> Regards,
>> 
>> Adolf.
>> 
>> On 13/05/2022 12:09, Michael Tremer wrote:
>>> Hello,
>>> 
>>>> On 12 May 2022, at 21:10, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>> 
>>>> Hi Michael,
>>>> 
>>>> On 12/05/2022 14:53, Michael Tremer wrote:
>>>>> Hello,
>>>>>> On 12 May 2022, at 12:25, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> On 12/05/2022 11:13, Michael Tremer wrote:
>>>>>>> Hello,
>>>>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
>>>>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
>>>>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
>>>>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
>>>>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
>>>>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
>>>>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
>>>>> Okay. That is indeed quite bad then.
>>>>> Since there is no kernel in 168, is there a chance that we broke the update to 167?
>>>> I don't know but as far as I have been concerned CU167 has worked well on my vm.
>>>> Is there something I should look for on my CU167 vm?
>>>>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
>>>>> Third time lucky isn’t good enough for me :)
>>>> Me neither :-)
>>>>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
>>>>>> 
>>>>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
>>>>> Have there been any modules loaded? If one loads, they all should load.
>>>> This is what I get with lsmod
>>>> 
>>>> Module            Size    Used by
>>>> crct10dif_pclmul    16384    1
>>>> crc32_pclmul        16384    0
>>>> ghash_clmulni_intel    16384    0
>>>> serio_raw        20480    0
>>>> ohci_pci        20480    0
>>>> ata_generic        16384    0
>>>> pata_acpi        16384    0
>>>> video            57344    0
>>> 
>>> It looks like these modules could have come from the initramdisk image. At least that is working well.
>>> 
>>>> On my running CU1678 vm lsmod gives the following
>>>> 
>>>> Module                  Size  Used by
>>>> tun                    61440  2
>>>> nfnetlink_queue        28672  1
>>>> xt_NFQUEUE             16384  8
>>>> xt_MASQUERADE          20480  1
>>>> cfg80211             1036288  0
>>>> rfkill                 32768  1 cfg80211
>>>> 8021q                  40960  0
>>>> garp                   16384  1 8021q
>>>> xt_set                 16384  260
>>>> ip_set_hash_net        49152  258
>>>> ip_set                 57344  2 xt_set,ip_set_hash_net
>>>> xt_hashlimit           20480  2
>>>> xt_multiport           20480  4
>>>> xt_policy              16384  5
>>>> xt_TCPMSS              16384  1
>>>> xt_conntrack           16384  7
>>>> xt_comment             16384  18
>>>> ipt_REJECT             16384  1
>>>> nf_reject_ipv4         16384  1 ipt_REJECT
>>>> xt_LOG                 20480  26
>>>> xt_limit               16384  25
>>>> xt_mark                16384  27
>>>> xt_connmark            16384  2
>>>> nf_log_syslog          24576  26
>>>> iptable_raw            16384  0
>>>> iptable_mangle         16384  1
>>>> iptable_filter         16384  1
>>>> vfat                   24576  1
>>>> fat                    90112  1 vfat
>>>> sch_cake               36864  4
>>>> intel_powerclamp       20480  0
>>>> psmouse               184320  0
>>>> pcspkr                 16384  0
>>>> i2c_piix4              28672  0
>>>> e1000                 163840  0
>>>> i2c_core              106496  2 psmouse,i2c_piix4
>>>> crct10dif_pclmul       16384  1
>>>> crc32_pclmul           16384  0
>>>> ata_generic            16384  0
>>>> pata_acpi              16384  0
>>>> ghash_clmulni_intel    16384  0
>>>> serio_raw              20480  0
>>>> ohci_pci               20480  0
>>>> video                  57344  0
>>> 
>>> Yes, this looks more like it.
>>> 
>>>> The last 8 entries are the same but a whole load of others are missing.
>>> 
>>> Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
>>> 
>>> I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
>>> 
>>> -Michael
>>> 
>>>> 
>>>> Regards,
>>>> Adolf.
>>>>>> Tried modprobe e1000 but this came back with the following error
>>>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service
>>>>>> 
>>>>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
>>>>>> 
>>>>>> Regards
>>>>>> Adolf
>>>>>>>> On 11 May 2022, at 20:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>> 
>>>>>>>> Hi All,
>>>>>>>> 
>>>>>>>> On 11/05/2022 20:48, Jon Murphy wrote:
>>>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
>>>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c*
>>>>>>>>> Shouldn’t this be Core Update 168?
>>>>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
>>>>>>>> 
>>>>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
>>>>>>>> 
>>>>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
>>>>>>>> 
>>>>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
>>>>>>>> 
>>>>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
>>>>>>> That seems to be a bug that should not be there. Did we recently update apache?
>>>>>>> -Michael
>>>>>>>> 
>>>>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
>>>>>>>> 
>>>>>>>> I think I am going to raise a bug on this.
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> Adolf.
>>>>>>>>> Jon
>>>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka(a)ipfire.org <mailto:adolf.belka(a)ipfire.org>> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi All,
>>>>>>>>>> 
>>>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote:
>>>>>>>>>>> Hi Leo,
>>>>>>>>>>> 
>>>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote:
>>>>>>>>>>>> Hi Adolf,
>>>>>>>>>>>> 
>>>>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
>>>>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
>>>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 <https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762>
>>>>>>>>>>> Now I see it in the code. Thanks very much.
>>>>>>>>>>>> 
>>>>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
>>>>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions.
>>>>>>>>>>> 
>>>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
>>>>>>>>>>> Will wait to hear other inputs on this problem.
>>>>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
>>>>>>>>>>> 
>>>>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
>>>>>>>>>> 
>>>>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
>>>>>>>>>> 
>>>>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> 
>>>>>>>>>> Adolf.
>>>>>>>>>> 
>>>>>>>>>>> Regards,
>>>>>>>>>>> Adolf.
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Best regards
>>>>>>>>>>>> Leo
>>>>>>>>>>>> 
>>>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> First try:-
>>>>>>>>>>>>> 
>>>>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>>>>>>>>>>>> 
>>>>>>>>>>>>> starting
>>>>>>>>>>>>> DUID 00:04:70:........
>>>>>>>>>>>>> red0: interface not found
>>>>>>>>>>>>> 
>>>>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Loading of module with unavailable key is rejected
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Second try:-
>>>>>>>>>>>>> 
>>>>>>>>>>>>> This time running the update took over 10 mins.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Repository was set back to Stable.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Before running the pakfire directory had the following
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>> 
>>>>>>>>>>>>> After running it had
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log
>>>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
>>>>>>>>>>>>> 
>>>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Rebooted.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>> 
>>>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Third try:-
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log
>>>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Rebooted
>>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>> Several repeats of the following two lines on the console screen:-
>>>>>>>>>>>>> 
>>>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>>>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I have given up at this point.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Adolf.
>>> 

Re: Feedback on problems with Core Update 168 Testing

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 3968 bytes --]

Hello,

> On 13 May 2022, at 16:02, Rob Brewer <ipfire-devel(a)grantura.co.uk> wrote:
> 
> On Thursday 12 May 2022 14:15 Rob Brewer wrote:
> 
>> On Thursday 12 May 2022 13:51 Michael Tremer wrote:
>> 
>>> Hello Rob,
>>> 
>>> You seem to already have lost the kernel there which is not part of 168.
>>> 
>>> Can you extract any log files from /var/log/pakfire?
>>> 
>>> -Michael
>>> 
>> I was upgrading 165-168 so I think Linux 5.15.23-ipfire was from 165.
>> I am trying to build a bootable USB drive so I can mount the ssd and look
>> at the Pakfire logs.
>> 
>> 
>> Rob
>> 
> I have been unable to boot from a usb drive so I re-fitted an old mSATA SSD 
> with an earlier version of IPFIRE and was surprisingly unable to boot from 
> there also.
> 
> On investigation the bios is reporting:
> 
> SeaBIOS (version rel-1.14.0.1-0-g8610266a)
> 
> yet I manually upgraded it to v4.16.0.2 a few weeks ago after a discussion 
> with Bernard. 
> 
> https://community.ipfire.org/t/entropy-with-new-apu-firmware/7707
> 
> Could something have downgraded or corrupted the BIOS?

No, we never touch this as it is too unpredictable what would happen.

> I'll try to re-flash back to v4.16.0.2 if I can and try to boot again.

But that is not the same version number that you are comparing there.

SeaBIOS is a part of the firmware and comes in a different version.

> 
> 
> Rob
> 
>>>> On 12 May 2022, at 11:43, Rob Brewer <ipfire-devel(a)grantura.co.uk>
>>>> wrote:
>>>> 
>>>> On Thursday 12 May 2022 10:13 Michael Tremer wrote:
>>>> 
>>>>> Hello,
>>>>> 
>>>>> Thanks for spending so much time on this. We definitely need to improve
>>>>> the general update experience since we sometimes seem to break people’s
>>>>> systems and it is not nice to re-install a firewall from scratch. It
>>>>> will take a while.
>>>>> 
>>>>> So what I can say is that the kernel module issues come from when the
>>>>> running kernel is changed and the kernel is trying to load any modules
>>>>> that now have changed. This fails by design, because we sign our kernel
>>>>> modules. The key is randomly generated at build time and used to sign
>>>>> all modules and it then thrown away. For each build, we are using a
>>>>> different, unique key that is not preserved.
>>>>> 
>>>>> This means that although the kernel modules are of the same version,
>>>>> they cannot be loaded because the signature check fails. That might
>>>>> also explain why you are seeing so many ipset errors, because the
>>>>> kernel cannot load that module any more. However, we use so much ipset
>>>>> now, why isn’t the module loaded from before the update was started?
>>>>> 
>>>>> The same goes for any network drivers. I assume you are using virtio or
>>>>> a generic e1000 network adapter which will have been initialised at
>>>>> boot time. The kernel should never unload the kernel module for that
>>>>> interface and load it again later. I have no idea what could have
>>>>> triggered that.
>>>>> 
>>>>> No matter what though; after you reboot, the new kernel should be
>>>>> booted being able to load all modules it wants and the system should
>>>>> run absolutely fine. Can you confirm that that is at least the case?
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> My Pakfire upgrade to 168 on my development APU2 board failed during
>>>> upgrade and I lost ethernet communication with the PC.
>>>> 
>>>> The APU2 now fails after the grub prompt with the error:
>>>> 
>>>> *IPFire 2.27 (x86_64) - core166 Development Build: master/8f696f60
>>>> GNU/Linu
>>>> 
>>>> Loading Linux 5.15.23-ipfire ...
>>>> error: file `/vmlinuz-5.15.23-ipfire' not found.
>>>> Loading initial ramdisk ...
>>>> error: you need to load the kernel first.
>>>> 
>>>> so it looks like update-initramfs didn't run after the upgrade.
>>>> 
>>>> I'll try to boot the box from a usbstick and see if I can access the
>>>> disk.
>>>> 
>>>> Rob

Re: Feedback on problems with Core Update 168 Testing

[Rob Brewer]

[-- Attachment #1: Type: text/plain, Size: 17246 bytes --]

On Friday 13 May 2022 16:02 Rob Brewer wrote:

> On Thursday 12 May 2022 14:15 Rob Brewer wrote:
> 
>> On Thursday 12 May 2022 13:51 Michael Tremer wrote:
>> 
>>> Hello Rob,
>>> 
>>> You seem to already have lost the kernel there which is not part of 168.
>>> 
>>> Can you extract any log files from /var/log/pakfire?
>>> 
>>> -Michael
>>>
>> I was upgrading 165-168 so I think Linux  5.15.23-ipfire was from 165.
>> I am trying to build a bootable USB drive so I can mount the ssd and look
>> at the Pakfire logs.
>> 
>> 
>> Rob
>>
> I have been unable to boot from a usb drive so I re-fitted an old mSATA
> SSD with an earlier version of IPFIRE and was surprisingly unable to boot
> from there also.
> 
> On investigation the bios is reporting:
> 
> SeaBIOS (version rel-1.14.0.1-0-g8610266a)
> 
> yet I manually upgraded it to  v4.16.0.2 a few weeks ago after a
> discussion with Bernard.
> 
> https://community.ipfire.org/t/entropy-with-new-apu-firmware/7707
> 
> Could something have downgraded or corrupted the BIOS?
> 
> I'll try to re-flash back to v4.16.0.2 if I can and try to boot again.
> 
> 
> Rob
>

Update..

Sucessfully booted from usb disk.

There appears to be an error in grub.cfg as the menu is looking for the wrong ipfire.img file:

/boot/grub/grub.cfg

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_cloud ###
### END /etc/grub.d/00_cloud ###

### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="${saved_entry}"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

serial --unit=0 --speed=115200
terminal_input serial
terminal_output serial
if [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.                              
else                              
  set timeout=5                                                    
fi                                        
### END /etc/grub.d/00_header ###         
                                          
### BEGIN /etc/grub.d/10_linux ###                                 
menuentry 'IPFire 2.27 (x86_64) - core166 Development Build: master/8f696f60 GNU/Linux' --class ipfire --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-1c836493-be8b-401c-ad0e-fb9ef6e85328' {
        load_video                                                 
        set gfxpayload=keep                                        
        insmod gzio                                                                                                                                                                                                          
        insmod part_msdos                 
        insmod ext2              
        set root='hd0,msdos1'             
        if [ x$feature_platform_search_hint = xy ]; then                                                                                                                                                                     
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  257d45b3-1483-44ab-815c-029ffa1bc8f6                                                           
        else                                                                                                                                                                                                                 
          search --no-floppy --fs-uuid --set=root 257d45b3-1483-44ab-815c-029ffa1bc8f6                                                                                                                                       
        fi                                                                                                                                                        
        echo    'Loading Linux 5.15.23-ipfire ...'
        linux   /vmlinuz-5.15.23-ipfire root=UUID=1c836493-be8b-401c-ad0e-fb9ef6e85328 ro panic=10 console=ttyS0,115200n8
        echo    'Loading initial ramdisk ...'           
        initrd  /initramfs-5.15.23-ipfire.img                                                                                                                                                                                
}                                                                                                                                                                 
submenu 'Advanced options for IPFire 2.27 (x86_64) - core166 Development Build: master/8f696f60 GNU/Linux' $menuentry_id_option 'gnulinux-advanced-1c836493-be8b-401c-ad0e-fb9ef6e85328' {                                   
        menuentry 'IPFire 2.27 (x86_64) - core166 Development Build: master/8f696f60 GNU/Linux, with Linux 5.15.23-ipfire' --class ipfire --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.15.23-ipfire-advan
                load_video                                                            
                set gfxpayload=keep                                                                                                                                                       
                insmod gzio                                                                                                                                                                                                           
                insmod part_msdos                                                                                        
                insmod ext2                                                                                                                                       
                set root='hd0,msdos1'                                                                                                                                                     
                if [ x$feature_platform_search_hint = xy ]; then                                                                                                                                                                      
                  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  257d45b3-1483-44ab-815c-029ffa1bc8f6                                                            
                else                                                                                                                                                                                                                  
                  search --no-floppy --fs-uuid --set=root 257d45b3-1483-44ab-815c-029ffa1bc8f6                                                                                                                                        
                fi                                                                                                                                                        
                echo    'Loading Linux 5.15.23-ipfire ...'                                                               
                linux   /vmlinuz-5.15.23-ipfire root=UUID=1c836493-be8b-401c-ad0e-fb9ef6e85328 ro panic=10 console=ttyS0,115200n8
                echo    'Loading initial ramdisk ...'                                                                                                                                     
                initrd  /initramfs-5.15.23-ipfire.img                                                                                                                                                                                 
        }                                                                                                                                                                                 
}                                                                                                                                                                                                                                     
                                                                                                                                                                          
### END /etc/grub.d/10_linux ###                                                              
                                                                                                                                                                                                                             
### BEGIN /etc/grub.d/20_linux_xen ###                                                                                                                                                                                       
                                                                                                                                 
### END /etc/grub.d/20_linux_xen ###                                                                                                                                      
                                                                
### BEGIN /etc/grub.d/30_os-prober ###                                                                                                                                                                                       
### END /etc/grub.d/30_os-prober ###                                                                                                                                                                                         
                                                                                              
### BEGIN /etc/grub.d/30_uefi-firmware ###                                                                                                                        
### END /etc/grub.d/30_uefi-firmware ###                                                                                                                          
                                                                                                                                 
### BEGIN /etc/grub.d/40_custom ###                                                   
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change                                                                                      
# the 'exec tail' line above.                                                                                                                                     
### END /etc/grub.d/40_custom ###                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                                                                                                                         
### BEGIN /etc/grub.d/41_custom ###                                                                                              
if [ -f  ${config_directory}/custom.cfg ]; then                                       
  source ${config_directory}/custom.cfg                                      
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then                                                                                                 
  source $prefix/custom.cfg                                                                                                                                       
fi                                                   

==================================

ls- -l /boot

-rw-r--r--    1 root     root       4869786 May  3 02:02 System.map-5.15.35-ipfire
-rw-r--r--    1 root     root        182471 May  3 02:02 config-5.15.35-ipfire
drwxr-xr-x    2 root     root          4096 Mar  1 12:55 efi
drwxr-xr-x    6 root     root          4096 May 11 07:59 grub
drwx------    2 root     root         16384 Mar  1 12:55 lost+found
-rw-r--r--    1 root     root       6993120 May  3 02:02 vmlinuz-5.15.35-ipfire


Grub should be looking for 5.15.35-ipfire but is incorrectly looking for 5.15.23-ipfire

========================================

root(a)box:~# ls -l /mnt/var/log/pakfire
total 1872
-rw-r--r--    1 root     root           422 May  7 09:29 install-cpufrequtils.log
-rw-r--r--    1 root     root            73 Mar 29 15:53 install-firmware-update.log
-rw-r--r--    1 root     root            66 Mar 29 15:53 install-flashrom.log
-rw-r--r--    1 root     root          1033 Mar 10 18:00 install-nano.log
-rw-r--r--    1 root     root           380 Mar 29 15:53 install-pcengines-apu-firmware.log
-rw-r--r--    1 root     root            79 Mar 29 15:52 install-rsync.log
-rw-r--r--    1 root     root        420549 Mar 29 12:14 update-core-upgrade-163.log
-rw-r--r--    1 root     root        289507 Mar 29 12:14 update-core-upgrade-164.log
-rw-r--r--    1 root     root        841876 May 11 07:59 update-core-upgrade-165.log
-rw-r--r--    1 root     root          1718 May 11 07:59 update-core-upgrade-166.log
-rw-r--r--    1 root     root        326352 May 11 08:00 update-core-upgrade-167.log
-rw-r--r--    1 root     root          2564 Mar 29 12:16 update-nano.log



Rob




   
>>>> On 12 May 2022, at 11:43, Rob Brewer <ipfire-devel(a)grantura.co.uk>
>>>> wrote:
>>>> 
>>>> On Thursday 12 May 2022 10:13 Michael Tremer wrote:
>>>> 
>>>>> Hello,
>>>>> 
>>>>> Thanks for spending so much time on this. We definitely need to
>>>>> improve the general update experience since we sometimes seem to break
>>>>> people’s systems and it is not nice to re-install a firewall from
>>>>> scratch. It will take a while.
>>>>> 
>>>>> So what I can say is that the kernel module issues come from when the
>>>>> running kernel is changed and the kernel is trying to load any modules
>>>>> that now have changed. This fails by design, because we sign our
>>>>> kernel modules. The key is randomly generated at build time and used
>>>>> to sign all modules and it then thrown away. For each build, we are
>>>>> using a different, unique key that is not preserved.
>>>>> 
>>>>> This means that although the kernel modules are of the same version,
>>>>> they cannot be loaded because the signature check fails. That might
>>>>> also explain why you are seeing so many ipset errors, because the
>>>>> kernel cannot load that module any more. However, we use so much ipset
>>>>> now, why isn’t the module loaded from before the update was started?
>>>>> 
>>>>> The same goes for any network drivers. I assume you are using virtio
>>>>> or a generic e1000 network adapter which will have been initialised at
>>>>> boot time. The kernel should never unload the kernel module for that
>>>>> interface and load it again later. I have no idea what could have
>>>>> triggered that.
>>>>> 
>>>>> No matter what though; after you reboot, the new kernel should be
>>>>> booted being able to load all modules it wants and the system should
>>>>> run absolutely fine. Can you confirm that that is at least the case?
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> My Pakfire upgrade to 168 on my development APU2 board failed during
>>>> upgrade and I lost ethernet communication with the PC.
>>>> 
>>>> The APU2 now fails after the grub prompt with the error:
>>>> 
>>>> *IPFire 2.27 (x86_64) - core166 Development Build: master/8f696f60
>>>> GNU/Linu
>>>> 
>>>> Loading Linux  5.15.23-ipfire ...
>>>> error: file `/vmlinuz-5.15.23-ipfire' not found.
>>>> Loading initial ramdisk ...
>>>> error: you need to load the kernel first.
>>>> 
>>>> so it looks like update-initramfs didn't run after the upgrade.
>>>> 
>>>> I'll try to boot the box from a usbstick and see if I can access the
>>>> disk.
>>>> 
>>>> Rob

Re: Feedback on problems with Core Update 168 Testing

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 27312 bytes --]

Hi All,

I think I may understand what is happening with my vm now. Of course I could also be wrong :-) but I will give it a go.

I saw bug#12862 where upgrading from CU166 to CU167 had stopped the raid array running so that IPFire was only on one disk.

I had missed seeing this in my evaluation of CU167 on my vm which does run with a raid array.

The vm shows the following

-bash-5.1$ cat /proc/mdstat
Personalities :
md127 : inactive sdb[1](S)
       52428724 blocks super 1.0

unused devices: <none>


It looks like IPFire was only running on sda.

So when I am doing an upgrade to CU168 it is likely that only one disk of the raid pair is being updated.

The CU168 vm that is working has the same as the above setting.

The CU168 vm that is not working has the following setting

-bash-5.1$ cat /proc/mdstat
Personalities :
md127 : inactive sda[0](S)
       52428724 blocks super 1.0

unused devices: <none>

So it looks like it has booted up on the disk that was not updated from the raid pair.


This probably also explains why occasionally I have had a vm that was not working suddenly start working properly when I reboot it, but booting again can put it back into non-working mode.

It is probably dependent on which of the two disks actually get used to boot from.

So the problem looks like it would only affect people who have raid setups and the problem originated in the CU167 upgrade.

So raid systems need to have their raid arrays fully working before doing an upgrade.

At least the above is my interpretation of what I have found. What does everyone else think.


Regards,

Adolf.


On 13/05/2022 15:59, Adolf Belka wrote:
> Hi Michael,
>
> I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.
>
> I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.
>
> However I did copy the boot directory info for these two cases and found an interesting effect.
>
> For the problem vm here was the boot directory layout of the CU167 base machine
>
> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
> drwx------  2 root root  16K Sep 30  2021 lost+found
> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>
> After upgrade and before reboot this had changed to
>
> drwxr-xr-x  5 root root 4.0K May 13 15:40 .
> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
> -rw-r--r--  1 root root 179K May  3 04:02 config-5.15.35-ipfire
> drwxr-xr-x  3 root root  16K Jan  1  1970 efi
> drwxr-xr-x  6 root root 4.0K May 13 15:41 grub
> -rw-------  1 root root  26M May 13 15:41 initramfs-5.15.35-ipfire.img
> drwx------  2 root root  16K Sep 30  2021 lost+found
> -rw-r--r--  1 root root 4.7M May  3 04:02 System.map-5.15.35-ipfire
> -rw-r--r--  1 root root 6.7M May  3 04:02 vmlinuz-5.15.35-ipfire
>
> Dates and times of some of the files/directories have changed as would be expected.
>
> After Reboot and with the problem the boot directory was now
>
> drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
> drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
> -rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
> drwxr-xr-x  3 root root 4.0K Sep 30  2021 efi
> drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
> -rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
> drwx------  2 root root  16K Sep 30  2021 lost+found
> -rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
> -rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire
>
> Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed
>
> For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.
>
> I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.
>
> Regards,
>
> Adolf.
>
> On 13/05/2022 12:09, Michael Tremer wrote:
>> Hello,
>>
>>> On 12 May 2022, at 21:10, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>
>>> Hi Michael,
>>>
>>> On 12/05/2022 14:53, Michael Tremer wrote:
>>>> Hello,
>>>>> On 12 May 2022, at 12:25, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> On 12/05/2022 11:13, Michael Tremer wrote:
>>>>>> Hello,
>>>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
>>>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
>>>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
>>>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
>>>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
>>>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
>>>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
>>>> Okay. That is indeed quite bad then.
>>>> Since there is no kernel in 168, is there a chance that we broke the update to 167?
>>> I don't know but as far as I have been concerned CU167 has worked well on my vm.
>>> Is there something I should look for on my CU167 vm?
>>>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
>>>> Third time lucky isn’t good enough for me :)
>>> Me neither :-)
>>>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
>>>>>
>>>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
>>>> Have there been any modules loaded? If one loads, they all should load.
>>> This is what I get with lsmod
>>>
>>> Module            Size    Used by
>>> crct10dif_pclmul    16384    1
>>> crc32_pclmul        16384    0
>>> ghash_clmulni_intel    16384    0
>>> serio_raw        20480    0
>>> ohci_pci        20480    0
>>> ata_generic        16384    0
>>> pata_acpi        16384    0
>>> video            57344    0
>>
>> It looks like these modules could have come from the initramdisk image. At least that is working well.
>>
>>> On my running CU1678 vm lsmod gives the following
>>>
>>> Module                  Size  Used by
>>> tun                    61440  2
>>> nfnetlink_queue        28672  1
>>> xt_NFQUEUE             16384  8
>>> xt_MASQUERADE          20480  1
>>> cfg80211             1036288  0
>>> rfkill                 32768  1 cfg80211
>>> 8021q                  40960  0
>>> garp                   16384  1 8021q
>>> xt_set                 16384  260
>>> ip_set_hash_net        49152  258
>>> ip_set                 57344  2 xt_set,ip_set_hash_net
>>> xt_hashlimit           20480  2
>>> xt_multiport           20480  4
>>> xt_policy              16384  5
>>> xt_TCPMSS              16384  1
>>> xt_conntrack           16384  7
>>> xt_comment             16384  18
>>> ipt_REJECT             16384  1
>>> nf_reject_ipv4         16384  1 ipt_REJECT
>>> xt_LOG                 20480  26
>>> xt_limit               16384  25
>>> xt_mark                16384  27
>>> xt_connmark            16384  2
>>> nf_log_syslog          24576  26
>>> iptable_raw            16384  0
>>> iptable_mangle         16384  1
>>> iptable_filter         16384  1
>>> vfat                   24576  1
>>> fat                    90112  1 vfat
>>> sch_cake               36864  4
>>> intel_powerclamp       20480  0
>>> psmouse               184320  0
>>> pcspkr                 16384  0
>>> i2c_piix4              28672  0
>>> e1000                 163840  0
>>> i2c_core              106496  2 psmouse,i2c_piix4
>>> crct10dif_pclmul       16384  1
>>> crc32_pclmul           16384  0
>>> ata_generic            16384  0
>>> pata_acpi              16384  0
>>> ghash_clmulni_intel    16384  0
>>> serio_raw              20480  0
>>> ohci_pci               20480  0
>>> video                  57344  0
>>
>> Yes, this looks more like it.
>>
>>> The last 8 entries are the same but a whole load of others are missing.
>>
>> Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
>>
>> I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
>>
>> -Michael
>>
>>>
>>> Regards,
>>> Adolf.
>>>>> Tried modprobe e1000 but this came back with the following error
>>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service
>>>>>
>>>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
>>>>>
>>>>> Regards
>>>>> Adolf
>>>>>>> On 11 May 2022, at 20:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> On 11/05/2022 20:48, Jon Murphy wrote:
>>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
>>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c*
>>>>>>>> Shouldn’t this be Core Update 168?
>>>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
>>>>>>>
>>>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
>>>>>>>
>>>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
>>>>>>>
>>>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
>>>>>>>
>>>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
>>>>>> That seems to be a bug that should not be there. Did we recently update apache?
>>>>>> -Michael
>>>>>>>
>>>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
>>>>>>>
>>>>>>> I think I am going to raise a bug on this.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Adolf.
>>>>>>>> Jon
>>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka(a)ipfire.org <mailto:adolf.belka(a)ipfire.org>> wrote:
>>>>>>>>>
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote:
>>>>>>>>>> Hi Leo,
>>>>>>>>>>
>>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote:
>>>>>>>>>>> Hi Adolf,
>>>>>>>>>>>
>>>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
>>>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
>>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 <https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762>
>>>>>>>>>> Now I see it in the code. Thanks very much.
>>>>>>>>>>>
>>>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
>>>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions.
>>>>>>>>>>
>>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
>>>>>>>>>> Will wait to hear other inputs on this problem.
>>>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
>>>>>>>>>>
>>>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
>>>>>>>>>
>>>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
>>>>>>>>>
>>>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Adolf.
>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Adolf.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Best regards
>>>>>>>>>>> Leo
>>>>>>>>>>>
>>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>
>>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>>>>>>>>>>>
>>>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> First try:-
>>>>>>>>>>>>
>>>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had
>>>>>>>>>>>>
>>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>
>>>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases.
>>>>>>>>>>>>
>>>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>>>>>>>>>>>
>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>>
>>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>>>>>>>>>>>
>>>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>>>>>>>>>>>
>>>>>>>>>>>> starting
>>>>>>>>>>>> DUID 00:04:70:........
>>>>>>>>>>>> red0: interface not found
>>>>>>>>>>>>
>>>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>>>>>>>>>>>
>>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same
>>>>>>>>>>>>
>>>>>>>>>>>> Loading of module with unavailable key is rejected
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Second try:-
>>>>>>>>>>>>
>>>>>>>>>>>> This time running the update took over 10 mins.
>>>>>>>>>>>>
>>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>>>>>>>>>>>
>>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>>>>>>>>>>>
>>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>>>>>>>>>>>
>>>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>>>>>>>>>>>
>>>>>>>>>>>> Repository was set back to Stable.
>>>>>>>>>>>>
>>>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>>>>>>>>>>>
>>>>>>>>>>>> Before running the pakfire directory had the following
>>>>>>>>>>>>
>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>
>>>>>>>>>>>> After running it had
>>>>>>>>>>>>
>>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log
>>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
>>>>>>>>>>>>
>>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>>>>>>>>>>>
>>>>>>>>>>>> Rebooted.
>>>>>>>>>>>>
>>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>>
>>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it
>>>>>>>>>>>>
>>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Third try:-
>>>>>>>>>>>>
>>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>>
>>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>>>>>>>>>>>
>>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log
>>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
>>>>>>>>>>>>
>>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>>>>>>>>>>>
>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>>>>>>>>>>>
>>>>>>>>>>>> Rebooted
>>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>> Several repeats of the following two lines on the console screen:-
>>>>>>>>>>>>
>>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I have given up at this point.
>>>>>>>>>>>>
>>>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>>>>>>>>>>>
>>>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Adolf.
>>

Re: Feedback on problems with Core Update 168 Testing

[Rob Brewer]

[-- Attachment #1: Type: text/plain, Size: 3446 bytes --]

On Thursday 12 May 2022 14:15 Rob Brewer wrote:

> On Thursday 12 May 2022 13:51 Michael Tremer wrote:
> 
>> Hello Rob,
>> 
>> You seem to already have lost the kernel there which is not part of 168.
>> 
>> Can you extract any log files from /var/log/pakfire?
>> 
>> -Michael
>>
> I was upgrading 165-168 so I think Linux  5.15.23-ipfire was from 165.
> I am trying to build a bootable USB drive so I can mount the ssd and look
> at the Pakfire logs.
> 
> 
> Rob
>
I have been unable to boot from a usb drive so I re-fitted an old mSATA SSD 
with an earlier version of IPFIRE and was surprisingly unable to boot from 
there also.

On investigation the bios is reporting:

SeaBIOS (version rel-1.14.0.1-0-g8610266a)

yet I manually upgraded it to  v4.16.0.2 a few weeks ago after a discussion 
with Bernard. 

https://community.ipfire.org/t/entropy-with-new-apu-firmware/7707

Could something have downgraded or corrupted the BIOS?

I'll try to re-flash back to v4.16.0.2 if I can and try to boot again.


Rob
  
>>> On 12 May 2022, at 11:43, Rob Brewer <ipfire-devel(a)grantura.co.uk>
>>> wrote:
>>> 
>>> On Thursday 12 May 2022 10:13 Michael Tremer wrote:
>>> 
>>>> Hello,
>>>> 
>>>> Thanks for spending so much time on this. We definitely need to improve
>>>> the general update experience since we sometimes seem to break people’s
>>>> systems and it is not nice to re-install a firewall from scratch. It
>>>> will take a while.
>>>> 
>>>> So what I can say is that the kernel module issues come from when the
>>>> running kernel is changed and the kernel is trying to load any modules
>>>> that now have changed. This fails by design, because we sign our kernel
>>>> modules. The key is randomly generated at build time and used to sign
>>>> all modules and it then thrown away. For each build, we are using a
>>>> different, unique key that is not preserved.
>>>> 
>>>> This means that although the kernel modules are of the same version,
>>>> they cannot be loaded because the signature check fails. That might
>>>> also explain why you are seeing so many ipset errors, because the
>>>> kernel cannot load that module any more. However, we use so much ipset
>>>> now, why isn’t the module loaded from before the update was started?
>>>> 
>>>> The same goes for any network drivers. I assume you are using virtio or
>>>> a generic e1000 network adapter which will have been initialised at
>>>> boot time. The kernel should never unload the kernel module for that
>>>> interface and load it again later. I have no idea what could have
>>>> triggered that.
>>>> 
>>>> No matter what though; after you reboot, the new kernel should be
>>>> booted being able to load all modules it wants and the system should
>>>> run absolutely fine. Can you confirm that that is at least the case?
>>>> 
>>> 
>>> 
>>> 
>>> My Pakfire upgrade to 168 on my development APU2 board failed during
>>> upgrade and I lost ethernet communication with the PC.
>>> 
>>> The APU2 now fails after the grub prompt with the error:
>>> 
>>> *IPFire 2.27 (x86_64) - core166 Development Build: master/8f696f60
>>> GNU/Linu
>>> 
>>> Loading Linux  5.15.23-ipfire ...
>>> error: file `/vmlinuz-5.15.23-ipfire' not found.
>>> Loading initial ramdisk ...
>>> error: you need to load the kernel first.
>>> 
>>> so it looks like update-initramfs didn't run after the upgrade.
>>> 
>>> I'll try to boot the box from a usbstick and see if I can access the
>>> disk.
>>> 
>>> Rob

Re: Feedback on problems with Core Update 168 Testing

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 24347 bytes --]

Hi Michael,

I have found some differences between the vm's that have upgraded with no problem and those that have lost most of the modules.

I did another three vm clones and upgrades and the first two all went with no problems. I was starting to think the issue had gone away but on the third one the same problem happened after the reboot.

However I did copy the boot directory info for these two cases and found an interesting effect.

For the problem vm here was the boot directory layout of the CU167 base machine

drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
-rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
drwxr-xr-x  3 root root  16K Jan  1  1970 efi
drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
-rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
drwx------  2 root root  16K Sep 30  2021 lost+found
-rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
-rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire

After upgrade and before reboot this had changed to

drwxr-xr-x  5 root root 4.0K May 13 15:40 .
drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
-rw-r--r--  1 root root 179K May  3 04:02 config-5.15.35-ipfire
drwxr-xr-x  3 root root  16K Jan  1  1970 efi
drwxr-xr-x  6 root root 4.0K May 13 15:41 grub
-rw-------  1 root root  26M May 13 15:41 initramfs-5.15.35-ipfire.img
drwx------  2 root root  16K Sep 30  2021 lost+found
-rw-r--r--  1 root root 4.7M May  3 04:02 System.map-5.15.35-ipfire
-rw-r--r--  1 root root 6.7M May  3 04:02 vmlinuz-5.15.35-ipfire

Dates and times of some of the files/directories have changed as would be expected.

After Reboot and with the problem the boot directory was now

drwxr-xr-x  5 root root 4.0K Apr 28 09:47 .
drwxr-xr-x 21 root root 4.0K Nov 28 15:49 ..
-rw-r--r--  1 root root 179K Apr 26 13:00 config-5.15.35-ipfire
drwxr-xr-x  3 root root 4.0K Sep 30  2021 efi
drwxr-xr-x  6 root root 4.0K Apr 28 09:47 grub
-rw-------  1 root root  26M Apr 28 09:47 initramfs-5.15.35-ipfire.img
drwx------  2 root root  16K Sep 30  2021 lost+found
-rw-r--r--  1 root root 4.7M Apr 26 13:00 System.map-5.15.35-ipfire
-rw-r--r--  1 root root 6.7M Apr 26 13:00 vmlinuz-5.15.35-ipfire

Which is the same as for the original CU167 machine except that the date/time of the efi directory has changed

For the vm clone that upgraded with no problems the boot directory after reboot was the same as after upgrade.

I don';t understand why or how this has happened but it is certainly a difference between the vm's having the problem and the ones not.

Regards,

Adolf.

On 13/05/2022 12:09, Michael Tremer wrote:
> Hello,
> 
>> On 12 May 2022, at 21:10, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi Michael,
>>
>> On 12/05/2022 14:53, Michael Tremer wrote:
>>> Hello,
>>>> On 12 May 2022, at 12:25, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>
>>>> Hi,
>>>>
>>>> On 12/05/2022 11:13, Michael Tremer wrote:
>>>>> Hello,
>>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
>>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
>>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
>>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
>>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
>>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
>>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
>>> Okay. That is indeed quite bad then.
>>> Since there is no kernel in 168, is there a chance that we broke the update to 167?
>> I don't know but as far as I have been concerned CU167 has worked well on my vm.
>> Is there something I should look for on my CU167 vm?
>>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
>>> Third time lucky isn’t good enough for me :)
>> Me neither :-)
>>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
>>>>
>>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
>>> Have there been any modules loaded? If one loads, they all should load.
>> This is what I get with lsmod
>>
>> Module			Size	Used by
>> crct10dif_pclmul	16384	1
>> crc32_pclmul		16384	0
>> ghash_clmulni_intel	16384	0
>> serio_raw		20480	0
>> ohci_pci		20480	0
>> ata_generic		16384	0
>> pata_acpi		16384	0
>> video			57344	0
> 
> It looks like these modules could have come from the initramdisk image. At least that is working well.
> 
>> On my running CU1678 vm lsmod gives the following
>>
>> Module                  Size  Used by
>> tun                    61440  2
>> nfnetlink_queue        28672  1
>> xt_NFQUEUE             16384  8
>> xt_MASQUERADE          20480  1
>> cfg80211             1036288  0
>> rfkill                 32768  1 cfg80211
>> 8021q                  40960  0
>> garp                   16384  1 8021q
>> xt_set                 16384  260
>> ip_set_hash_net        49152  258
>> ip_set                 57344  2 xt_set,ip_set_hash_net
>> xt_hashlimit           20480  2
>> xt_multiport           20480  4
>> xt_policy              16384  5
>> xt_TCPMSS              16384  1
>> xt_conntrack           16384  7
>> xt_comment             16384  18
>> ipt_REJECT             16384  1
>> nf_reject_ipv4         16384  1 ipt_REJECT
>> xt_LOG                 20480  26
>> xt_limit               16384  25
>> xt_mark                16384  27
>> xt_connmark            16384  2
>> nf_log_syslog          24576  26
>> iptable_raw            16384  0
>> iptable_mangle         16384  1
>> iptable_filter         16384  1
>> vfat                   24576  1
>> fat                    90112  1 vfat
>> sch_cake               36864  4
>> intel_powerclamp       20480  0
>> psmouse               184320  0
>> pcspkr                 16384  0
>> i2c_piix4              28672  0
>> e1000                 163840  0
>> i2c_core              106496  2 psmouse,i2c_piix4
>> crct10dif_pclmul       16384  1
>> crc32_pclmul           16384  0
>> ata_generic            16384  0
>> pata_acpi              16384  0
>> ghash_clmulni_intel    16384  0
>> serio_raw              20480  0
>> ohci_pci               20480  0
>> video                  57344  0
> 
> Yes, this looks more like it.
> 
>> The last 8 entries are the same but a whole load of others are missing.
> 
> Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.
> 
> I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.
> 
> -Michael
> 
>>
>> Regards,
>> Adolf.
>>>> Tried modprobe e1000 but this came back with the following error
>>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service
>>>>
>>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
>>>>
>>>> Regards
>>>> Adolf
>>>>>> On 11 May 2022, at 20:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> On 11/05/2022 20:48, Jon Murphy wrote:
>>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
>>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c*
>>>>>>> Shouldn’t this be Core Update 168?
>>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
>>>>>>
>>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
>>>>>>
>>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
>>>>>>
>>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
>>>>>>
>>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
>>>>> That seems to be a bug that should not be there. Did we recently update apache?
>>>>> -Michael
>>>>>>
>>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
>>>>>>
>>>>>> I think I am going to raise a bug on this.
>>>>>>
>>>>>> Regards,
>>>>>> Adolf.
>>>>>>> Jon
>>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka(a)ipfire.org <mailto:adolf.belka(a)ipfire.org>> wrote:
>>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote:
>>>>>>>>> Hi Leo,
>>>>>>>>>
>>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote:
>>>>>>>>>> Hi Adolf,
>>>>>>>>>>
>>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
>>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 <https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762>
>>>>>>>>> Now I see it in the code. Thanks very much.
>>>>>>>>>>
>>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
>>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions.
>>>>>>>>>
>>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
>>>>>>>>> Will wait to hear other inputs on this problem.
>>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
>>>>>>>>>
>>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
>>>>>>>>
>>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
>>>>>>>>
>>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Adolf.
>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Adolf.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Best regards
>>>>>>>>>> Leo
>>>>>>>>>>
>>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>>>>>>>>>>> Hi All,
>>>>>>>>>>>
>>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>>>>>>>>>>
>>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> First try:-
>>>>>>>>>>>
>>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had
>>>>>>>>>>>
>>>>>>>>>>> Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>
>>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases.
>>>>>>>>>>>
>>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>>>>>>>>>>
>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>>
>>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>>>>>>>>>>
>>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>>>>>>>>>>
>>>>>>>>>>> starting
>>>>>>>>>>> DUID 00:04:70:........
>>>>>>>>>>> red0: interface not found
>>>>>>>>>>>
>>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>>>>>>>>>>
>>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same
>>>>>>>>>>>
>>>>>>>>>>> Loading of module with unavailable key is rejected
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Second try:-
>>>>>>>>>>>
>>>>>>>>>>> This time running the update took over 10 mins.
>>>>>>>>>>>
>>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>>>>>>>>>>
>>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>>>>>>>>>>
>>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>>>>>>>>>>
>>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>>>>>>>>>>
>>>>>>>>>>> Repository was set back to Stable.
>>>>>>>>>>>
>>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>>>>>>>>>>
>>>>>>>>>>> Before running the pakfire directory had the following
>>>>>>>>>>>
>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>
>>>>>>>>>>> After running it had
>>>>>>>>>>>
>>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log
>>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
>>>>>>>>>>>
>>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>>>>>>>>>>
>>>>>>>>>>> Rebooted.
>>>>>>>>>>>
>>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>>>>>>>>>>
>>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it
>>>>>>>>>>>
>>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Third try:-
>>>>>>>>>>>
>>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>>>
>>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>>>>>>>>>>
>>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log
>>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
>>>>>>>>>>>
>>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>>>>>>>>>>
>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>>>>>>>>>>
>>>>>>>>>>> Rebooted
>>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>>> Several repeats of the following two lines on the console screen:-
>>>>>>>>>>>
>>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I have given up at this point.
>>>>>>>>>>>
>>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>>>>>>>>>>
>>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>> Adolf.
> 

Re: Feedback on problems with Core Update 168 Testing

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 21356 bytes --]

Hello,

> On 12 May 2022, at 21:10, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi Michael,
> 
> On 12/05/2022 14:53, Michael Tremer wrote:
>> Hello,
>>> On 12 May 2022, at 12:25, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>> 
>>> Hi,
>>> 
>>> On 12/05/2022 11:13, Michael Tremer wrote:
>>>> Hello,
>>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
>>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
>>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
>>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
>>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
>>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
>>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
>> Okay. That is indeed quite bad then.
>> Since there is no kernel in 168, is there a chance that we broke the update to 167?
> I don't know but as far as I have been concerned CU167 has worked well on my vm.
> Is there something I should look for on my CU167 vm?
>>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
>> Third time lucky isn’t good enough for me :)
> Me neither :-)
>>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
>>> 
>>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
>> Have there been any modules loaded? If one loads, they all should load.
> This is what I get with lsmod
> 
> Module			Size	Used by
> crct10dif_pclmul	16384	1
> crc32_pclmul		16384	0
> ghash_clmulni_intel	16384	0
> serio_raw		20480	0
> ohci_pci		20480	0
> ata_generic		16384	0
> pata_acpi		16384	0
> video			57344	0

It looks like these modules could have come from the initramdisk image. At least that is working well.

> On my running CU1678 vm lsmod gives the following
> 
> Module                  Size  Used by
> tun                    61440  2
> nfnetlink_queue        28672  1
> xt_NFQUEUE             16384  8
> xt_MASQUERADE          20480  1
> cfg80211             1036288  0
> rfkill                 32768  1 cfg80211
> 8021q                  40960  0
> garp                   16384  1 8021q
> xt_set                 16384  260
> ip_set_hash_net        49152  258
> ip_set                 57344  2 xt_set,ip_set_hash_net
> xt_hashlimit           20480  2
> xt_multiport           20480  4
> xt_policy              16384  5
> xt_TCPMSS              16384  1
> xt_conntrack           16384  7
> xt_comment             16384  18
> ipt_REJECT             16384  1
> nf_reject_ipv4         16384  1 ipt_REJECT
> xt_LOG                 20480  26
> xt_limit               16384  25
> xt_mark                16384  27
> xt_connmark            16384  2
> nf_log_syslog          24576  26
> iptable_raw            16384  0
> iptable_mangle         16384  1
> iptable_filter         16384  1
> vfat                   24576  1
> fat                    90112  1 vfat
> sch_cake               36864  4
> intel_powerclamp       20480  0
> psmouse               184320  0
> pcspkr                 16384  0
> i2c_piix4              28672  0
> e1000                 163840  0
> i2c_core              106496  2 psmouse,i2c_piix4
> crct10dif_pclmul       16384  1
> crc32_pclmul           16384  0
> ata_generic            16384  0
> pata_acpi              16384  0
> ghash_clmulni_intel    16384  0
> serio_raw              20480  0
> ohci_pci               20480  0
> video                  57344  0

Yes, this looks more like it.

> The last 8 entries are the same but a whole load of others are missing.

Stefan gave me a call this morning telling me that he tried to reproduce this but he couldn’t.

I am not sure what we can do here. We urgently need to fix this since we might break people’s installations. But something seems to work differently for Adolf than for everybody else.

-Michael

> 
> Regards,
> Adolf.
>>> Tried modprobe e1000 but this came back with the following error
>>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service
>>> 
>>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
>>> 
>>> Regards
>>> Adolf
>>>>> On 11 May 2022, at 20:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>> 
>>>>> Hi All,
>>>>> 
>>>>> On 11/05/2022 20:48, Jon Murphy wrote:
>>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
>>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c*
>>>>>> Shouldn’t this be Core Update 168?
>>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
>>>>> 
>>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
>>>>> 
>>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
>>>>> 
>>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
>>>>> 
>>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
>>>> That seems to be a bug that should not be there. Did we recently update apache?
>>>> -Michael
>>>>> 
>>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
>>>>> 
>>>>> I think I am going to raise a bug on this.
>>>>> 
>>>>> Regards,
>>>>> Adolf.
>>>>>> Jon
>>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka(a)ipfire.org <mailto:adolf.belka(a)ipfire.org>> wrote:
>>>>>>> 
>>>>>>> Hi All,
>>>>>>> 
>>>>>>> On 11/05/2022 16:00, Adolf Belka wrote:
>>>>>>>> Hi Leo,
>>>>>>>> 
>>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote:
>>>>>>>>> Hi Adolf,
>>>>>>>>> 
>>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
>>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 <https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762>
>>>>>>>> Now I see it in the code. Thanks very much.
>>>>>>>>> 
>>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
>>>>>>>> Thanks for your help anyway. It's got rid of one of my questions.
>>>>>>>> 
>>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
>>>>>>>> Will wait to hear other inputs on this problem.
>>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
>>>>>>>> 
>>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
>>>>>>> 
>>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
>>>>>>> 
>>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
>>>>>>> 
>>>>>>> Regards,
>>>>>>> 
>>>>>>> Adolf.
>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> Adolf.
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Best regards
>>>>>>>>> Leo
>>>>>>>>> 
>>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>>>>>>>>>> Hi All,
>>>>>>>>>> 
>>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>>>>>>>>> 
>>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> First try:-
>>>>>>>>>> 
>>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had
>>>>>>>>>> 
>>>>>>>>>> Core Update 167 Development Build: master/c22d834c
>>>>>>>>>> 
>>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases.
>>>>>>>>>> 
>>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>>>>>>>>> 
>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>> 
>>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>>>>>>>>> 
>>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>>>>>>>>> 
>>>>>>>>>> starting
>>>>>>>>>> DUID 00:04:70:........
>>>>>>>>>> red0: interface not found
>>>>>>>>>> 
>>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>>>>>>>>> 
>>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same
>>>>>>>>>> 
>>>>>>>>>> Loading of module with unavailable key is rejected
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Second try:-
>>>>>>>>>> 
>>>>>>>>>> This time running the update took over 10 mins.
>>>>>>>>>> 
>>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>>>>>>>>> 
>>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>>>>>>>>> 
>>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>>>>>>>>> 
>>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>>>>>>>>> 
>>>>>>>>>> Repository was set back to Stable.
>>>>>>>>>> 
>>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>>>>>>>>> 
>>>>>>>>>> Before running the pakfire directory had the following
>>>>>>>>>> 
>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>> 
>>>>>>>>>> After running it had
>>>>>>>>>> 
>>>>>>>>>> -rw-r--r-- 1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 13:24 update-core-upgrade-167.log
>>>>>>>>>> -rw-r--r-- 1 root root 175 May 11 13:24 update-core-upgrade-168.log
>>>>>>>>>> 
>>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>>>>>>>>> 
>>>>>>>>>> Rebooted.
>>>>>>>>>> 
>>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>>>>>>>>> 
>>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it
>>>>>>>>>> 
>>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Third try:-
>>>>>>>>>> 
>>>>>>>>>> -rw-r--r-- 1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>> 
>>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>>>>>>>>> 
>>>>>>>>>> -rw-r--r-- 1 root root 604K May 11 14:00 update-core-upgrade-167.log
>>>>>>>>>> -rw-r--r-- 1 root root 63K May 11 14:01 update-core-upgrade-168.log
>>>>>>>>>> 
>>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>>>>>>>>> 
>>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>>>>>>>>> 
>>>>>>>>>> Rebooted
>>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>> Several repeats of the following two lines on the console screen:-
>>>>>>>>>> 
>>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> I have given up at this point.
>>>>>>>>>> 
>>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>>>>>>>>> 
>>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> 
>>>>>>>>>> Adolf.

Re: Feedback on problems with Core Update 168 Testing

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 20401 bytes --]

Hi Michael,

On 12/05/2022 14:53, Michael Tremer wrote:
> Hello,
> 
>> On 12 May 2022, at 12:25, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi,
>>
>> On 12/05/2022 11:13, Michael Tremer wrote:
>>> Hello,
>>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
>>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
>>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
>>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
>> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
>>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
>> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.
> 
> Okay. That is indeed quite bad then.
> 
> Since there is no kernel in 168, is there a chance that we broke the update to 167?
I don't know but as far as I have been concerned CU167 has worked well on my vm.
Is there something I should look for on my CU167 vm?
> 
>> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.
> 
> Third time lucky isn’t good enough for me :)
Me neither :-)
> 
>> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
>>
>> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers
> 
> Have there been any modules loaded? If one loads, they all should load.
This is what I get with lsmod

Module			Size	Used by
crct10dif_pclmul	16384	1
crc32_pclmul		16384	0
ghash_clmulni_intel	16384	0
serio_raw		20480	0
ohci_pci		20480	0
ata_generic		16384	0
pata_acpi		16384	0
video			57344	0

On my running CU1678 vm lsmod gives the following

Module                  Size  Used by
tun                    61440  2
nfnetlink_queue        28672  1
xt_NFQUEUE             16384  8
xt_MASQUERADE          20480  1
cfg80211             1036288  0
rfkill                 32768  1 cfg80211
8021q                  40960  0
garp                   16384  1 8021q
xt_set                 16384  260
ip_set_hash_net        49152  258
ip_set                 57344  2 xt_set,ip_set_hash_net
xt_hashlimit           20480  2
xt_multiport           20480  4
xt_policy              16384  5
xt_TCPMSS              16384  1
xt_conntrack           16384  7
xt_comment             16384  18
ipt_REJECT             16384  1
nf_reject_ipv4         16384  1 ipt_REJECT
xt_LOG                 20480  26
xt_limit               16384  25
xt_mark                16384  27
xt_connmark            16384  2
nf_log_syslog          24576  26
iptable_raw            16384  0
iptable_mangle         16384  1
iptable_filter         16384  1
vfat                   24576  1
fat                    90112  1 vfat
sch_cake               36864  4
intel_powerclamp       20480  0
psmouse               184320  0
pcspkr                 16384  0
i2c_piix4              28672  0
e1000                 163840  0
i2c_core              106496  2 psmouse,i2c_piix4
crct10dif_pclmul       16384  1
crc32_pclmul           16384  0
ata_generic            16384  0
pata_acpi              16384  0
ghash_clmulni_intel    16384  0
serio_raw              20480  0
ohci_pci               20480  0
video                  57344  0

The last 8 entries are the same but a whole load of others are missing.

Regards,
Adolf.
> 
>> Tried modprobe e1000 but this came back with the following error
>> modprobe: ERROR: could not insert 'e1000': Key was rejected by service
>>
>> So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
>>
>> Regards
>> Adolf
>>>> On 11 May 2022, at 20:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>>>
>>>> Hi All,
>>>>
>>>> On 11/05/2022 20:48, Jon Murphy wrote:
>>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
>>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c*
>>>>> Shouldn’t this be Core Update 168?
>>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
>>>>
>>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
>>>>
>>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
>>>>
>>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
>>>>
>>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
>>> That seems to be a bug that should not be there. Did we recently update apache?
>>> -Michael
>>>>
>>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
>>>>
>>>> I think I am going to raise a bug on this.
>>>>
>>>> Regards,
>>>> Adolf.
>>>>> Jon
>>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka(a)ipfire.org <mailto:adolf.belka(a)ipfire.org>> wrote:
>>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> On 11/05/2022 16:00, Adolf Belka wrote:
>>>>>>> Hi Leo,
>>>>>>>
>>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote:
>>>>>>>> Hi Adolf,
>>>>>>>>
>>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
>>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 <https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762>
>>>>>>> Now I see it in the code. Thanks very much.
>>>>>>>>
>>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
>>>>>>> Thanks for your help anyway. It's got rid of one of my questions.
>>>>>>>
>>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
>>>>>>> Will wait to hear other inputs on this problem.
>>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
>>>>>>>
>>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
>>>>>>
>>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
>>>>>>
>>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Adolf.
>>>>>>
>>>>>>> Regards,
>>>>>>> Adolf.
>>>>>>>
>>>>>>>>
>>>>>>>> Best regards
>>>>>>>> Leo
>>>>>>>>
>>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>>>>>>>>
>>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> First try:-
>>>>>>>>>
>>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had
>>>>>>>>>
>>>>>>>>> Core Update 167 Development Build: master/c22d834c
>>>>>>>>>
>>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases.
>>>>>>>>>
>>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>>>>>>>>
>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>>
>>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>>>>>>>>
>>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>>>>>>>>
>>>>>>>>> starting
>>>>>>>>> DUID 00:04:70:........
>>>>>>>>> red0: interface not found
>>>>>>>>>
>>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>>>>>>>>
>>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same
>>>>>>>>>
>>>>>>>>> Loading of module with unavailable key is rejected
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Second try:-
>>>>>>>>>
>>>>>>>>> This time running the update took over 10 mins.
>>>>>>>>>
>>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>>>>>>>>
>>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>>>>>>>>
>>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>>>>>>>>
>>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>>>>>>>>
>>>>>>>>> Repository was set back to Stable.
>>>>>>>>>
>>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>>>>>>>>
>>>>>>>>> Before running the pakfire directory had the following
>>>>>>>>>
>>>>>>>>> -rw-r--r--  1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>> -rw-r--r--  1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>
>>>>>>>>> After running it had
>>>>>>>>>
>>>>>>>>> -rw-r--r--  1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>>> -rw-r--r--  1 root root 604K May 11 13:24 update-core-upgrade-167.log
>>>>>>>>> -rw-r--r--  1 root root  175 May 11 13:24 update-core-upgrade-168.log
>>>>>>>>>
>>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>>>>>>>>
>>>>>>>>> Rebooted.
>>>>>>>>>
>>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>>>>>>>>
>>>>>>>>> /opt/pakfire/db/core/mine has 168 in it
>>>>>>>>>
>>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Third try:-
>>>>>>>>>
>>>>>>>>> -rw-r--r--  1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>>>
>>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>>>>>>>>
>>>>>>>>> -rw-r--r--  1 root root 604K May 11 14:00 update-core-upgrade-167.log
>>>>>>>>> -rw-r--r--  1 root root  63K May 11 14:01 update-core-upgrade-168.log
>>>>>>>>>
>>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>>>>>>>>
>>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>>>>>>>>
>>>>>>>>> Rebooted
>>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>>> Several repeats of the following two lines on the console screen:-
>>>>>>>>>
>>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>>>>>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have given up at this point.
>>>>>>>>>
>>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>>>>>>>>
>>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Adolf.
>>>>>>>>>
> 

Re: Feedback on problems with Core Update 168 Testing

[Rob Brewer]

[-- Attachment #1: Type: text/plain, Size: 2767 bytes --]

On Thursday 12 May 2022 13:51 Michael Tremer wrote:

> Hello Rob,
> 
> You seem to already have lost the kernel there which is not part of 168.
> 
> Can you extract any log files from /var/log/pakfire?
> 
> -Michael
>
I was upgrading 165-168 so I think Linux  5.15.23-ipfire was from 165.
I am trying to build a bootable USB drive so I can mount the ssd and look at 
the Pakfire logs.


Rob
 
>> On 12 May 2022, at 11:43, Rob Brewer <ipfire-devel(a)grantura.co.uk> wrote:
>> 
>> On Thursday 12 May 2022 10:13 Michael Tremer wrote:
>> 
>>> Hello,
>>> 
>>> Thanks for spending so much time on this. We definitely need to improve
>>> the general update experience since we sometimes seem to break people’s
>>> systems and it is not nice to re-install a firewall from scratch. It
>>> will take a while.
>>> 
>>> So what I can say is that the kernel module issues come from when the
>>> running kernel is changed and the kernel is trying to load any modules
>>> that now have changed. This fails by design, because we sign our kernel
>>> modules. The key is randomly generated at build time and used to sign
>>> all modules and it then thrown away. For each build, we are using a
>>> different, unique key that is not preserved.
>>> 
>>> This means that although the kernel modules are of the same version,
>>> they cannot be loaded because the signature check fails. That might also
>>> explain why you are seeing so many ipset errors, because the kernel
>>> cannot load that module any more. However, we use so much ipset now, why
>>> isn’t the module loaded from before the update was started?
>>> 
>>> The same goes for any network drivers. I assume you are using virtio or
>>> a generic e1000 network adapter which will have been initialised at boot
>>> time. The kernel should never unload the kernel module for that
>>> interface and load it again later. I have no idea what could have
>>> triggered that.
>>> 
>>> No matter what though; after you reboot, the new kernel should be booted
>>> being able to load all modules it wants and the system should run
>>> absolutely fine. Can you confirm that that is at least the case?
>>> 
>> 
>> 
>> 
>> My Pakfire upgrade to 168 on my development APU2 board failed during
>> upgrade and I lost ethernet communication with the PC.
>> 
>> The APU2 now fails after the grub prompt with the error:
>> 
>> *IPFire 2.27 (x86_64) - core166 Development Build: master/8f696f60
>> GNU/Linu
>> 
>> Loading Linux  5.15.23-ipfire ...
>> error: file `/vmlinuz-5.15.23-ipfire' not found.
>> Loading initial ramdisk ...
>> error: you need to load the kernel first.
>> 
>> so it looks like update-initramfs didn't run after the upgrade.
>> 
>> I'll try to boot the box from a usbstick and see if I can access the
>> disk.
>> 
>> Rob

Re: Feedback on problems with Core Update 168 Testing

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 18032 bytes --]

Hello,

> On 12 May 2022, at 12:25, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi,
> 
> On 12/05/2022 11:13, Michael Tremer wrote:
>> Hello,
>> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
>> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
>> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
>> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
> It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
>> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
> No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.

Okay. That is indeed quite bad then.

Since there is no kernel in 168, is there a chance that we broke the update to 167?

> Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.

Third time lucky isn’t good enough for me :)

> I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.
> 
> I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers

Have there been any modules loaded? If one loads, they all should load.

> Tried modprobe e1000 but this came back with the following error
> modprobe: ERROR: could not insert 'e1000': Key was rejected by service
> 
> So out of 7 or 8 reboots the vm booted with the network drivers loaded once.
> 
> Regards
> Adolf
>>> On 11 May 2022, at 20:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>> 
>>> Hi All,
>>> 
>>> On 11/05/2022 20:48, Jon Murphy wrote:
>>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
>>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c*
>>>> Shouldn’t this be Core Update 168?
>>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
>>> 
>>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
>>> 
>>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
>>> 
>>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
>>> 
>>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
>> That seems to be a bug that should not be there. Did we recently update apache?
>> -Michael
>>> 
>>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
>>> 
>>> I think I am going to raise a bug on this.
>>> 
>>> Regards,
>>> Adolf.
>>>> Jon
>>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka(a)ipfire.org <mailto:adolf.belka(a)ipfire.org>> wrote:
>>>>> 
>>>>> Hi All,
>>>>> 
>>>>> On 11/05/2022 16:00, Adolf Belka wrote:
>>>>>> Hi Leo,
>>>>>> 
>>>>>> On 11/05/2022 15:26, Leo Hofmann wrote:
>>>>>>> Hi Adolf,
>>>>>>> 
>>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
>>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 <https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762>
>>>>>> Now I see it in the code. Thanks very much.
>>>>>>> 
>>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
>>>>>> Thanks for your help anyway. It's got rid of one of my questions.
>>>>>> 
>>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
>>>>>> Will wait to hear other inputs on this problem.
>>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
>>>>>> 
>>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
>>>>> 
>>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
>>>>> 
>>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> Adolf.
>>>>> 
>>>>>> Regards,
>>>>>> Adolf.
>>>>>> 
>>>>>>> 
>>>>>>> Best regards
>>>>>>> Leo
>>>>>>> 
>>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>>>>>>>> Hi All,
>>>>>>>> 
>>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>>>>>>> 
>>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> First try:-
>>>>>>>> 
>>>>>>>> After running the update, which went quite quickly, the bottom of the screen had
>>>>>>>> 
>>>>>>>> Core Update 167 Development Build: master/c22d834c
>>>>>>>> 
>>>>>>>> I couldn't remember if this is what is expected or not for Testing releases.
>>>>>>>> 
>>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>>>>>>> 
>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>> 
>>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>>>>>>> 
>>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>>>>>>> 
>>>>>>>> starting
>>>>>>>> DUID 00:04:70:........
>>>>>>>> red0: interface not found
>>>>>>>> 
>>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>>>>>>> 
>>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same
>>>>>>>> 
>>>>>>>> Loading of module with unavailable key is rejected
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Second try:-
>>>>>>>> 
>>>>>>>> This time running the update took over 10 mins.
>>>>>>>> 
>>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>>>>>>> 
>>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>>>>>>> 
>>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>>>>>>> 
>>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>>>>>>> 
>>>>>>>> Repository was set back to Stable.
>>>>>>>> 
>>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>>>>>>> 
>>>>>>>> Before running the pakfire directory had the following
>>>>>>>> 
>>>>>>>> -rw-r--r--  1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>> -rw-r--r--  1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>> 
>>>>>>>> After running it had
>>>>>>>> 
>>>>>>>> -rw-r--r--  1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>>> -rw-r--r--  1 root root 604K May 11 13:24 update-core-upgrade-167.log
>>>>>>>> -rw-r--r--  1 root root  175 May 11 13:24 update-core-upgrade-168.log
>>>>>>>> 
>>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>>>>>>> 
>>>>>>>> Rebooted.
>>>>>>>> 
>>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>>>>>>> 
>>>>>>>> /opt/pakfire/db/core/mine has 168 in it
>>>>>>>> 
>>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Third try:-
>>>>>>>> 
>>>>>>>> -rw-r--r--  1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>> 
>>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>>>>>>> 
>>>>>>>> -rw-r--r--  1 root root 604K May 11 14:00 update-core-upgrade-167.log
>>>>>>>> -rw-r--r--  1 root root  63K May 11 14:01 update-core-upgrade-168.log
>>>>>>>> 
>>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>>>>>>> 
>>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>>>>>>> 
>>>>>>>> 
>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>>>>>>> 
>>>>>>>> Rebooted
>>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>> Several repeats of the following two lines on the console screen:-
>>>>>>>> 
>>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>>>>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> I have given up at this point.
>>>>>>>> 
>>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>>>>>>> 
>>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> 
>>>>>>>> Adolf.
>>>>>>>> 

Re: Feedback on problems with Core Update 168 Testing

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 2524 bytes --]

Hello Rob,

You seem to already have lost the kernel there which is not part of 168.

Can you extract any log files from /var/log/pakfire?

-Michael

> On 12 May 2022, at 11:43, Rob Brewer <ipfire-devel(a)grantura.co.uk> wrote:
> 
> On Thursday 12 May 2022 10:13 Michael Tremer wrote:
> 
>> Hello,
>> 
>> Thanks for spending so much time on this. We definitely need to improve
>> the general update experience since we sometimes seem to break people’s
>> systems and it is not nice to re-install a firewall from scratch. It will
>> take a while.
>> 
>> So what I can say is that the kernel module issues come from when the
>> running kernel is changed and the kernel is trying to load any modules
>> that now have changed. This fails by design, because we sign our kernel
>> modules. The key is randomly generated at build time and used to sign all
>> modules and it then thrown away. For each build, we are using a different,
>> unique key that is not preserved.
>> 
>> This means that although the kernel modules are of the same version, they
>> cannot be loaded because the signature check fails. That might also
>> explain why you are seeing so many ipset errors, because the kernel cannot
>> load that module any more. However, we use so much ipset now, why isn’t
>> the module loaded from before the update was started?
>> 
>> The same goes for any network drivers. I assume you are using virtio or a
>> generic e1000 network adapter which will have been initialised at boot
>> time. The kernel should never unload the kernel module for that interface
>> and load it again later. I have no idea what could have triggered that.
>> 
>> No matter what though; after you reboot, the new kernel should be booted
>> being able to load all modules it wants and the system should run
>> absolutely fine. Can you confirm that that is at least the case?
>> 
> 
> 
> 
> My Pakfire upgrade to 168 on my development APU2 board failed during upgrade 
> and I lost ethernet communication with the PC.
> 
> The APU2 now fails after the grub prompt with the error: 
> 
> *IPFire 2.27 (x86_64) - core166 Development Build: master/8f696f60 GNU/Linu
> 
> Loading Linux 5.15.23-ipfire ...
> error: file `/vmlinuz-5.15.23-ipfire' not found.
> Loading initial ramdisk ...
> error: you need to load the kernel first.
> 
> so it looks like update-initramfs didn't run after the upgrade.
> 
> I'll try to boot the box from a usbstick and see if I can access the disk.
> 
> Rob

Re: Feedback on problems with Core Update 168 Testing

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 17380 bytes --]

Hi,

On 12/05/2022 11:13, Michael Tremer wrote:
> Hello,
> 
> Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.
> 
> So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.
> 
> This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?
> 
> The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.
It is a generic e1000 network adapter that is being used by VirtualBox. Confirmed that on my working CU167 vm.
> 
> No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?
No it is not the case. Yesterday when I booted the vm several times the same error messages occurred each time and I always had no network interfaces.

Just now I tried booting my third attempt vm again and this time it ran okay and I ended up with all my networks assigned and I had network connection. An IP was assigned by dhcp to the red interface.

I then rebooted again and this time the error messages were back. I rebooted the vm three more times and each time the error messages were there and no network interfaces.

I checked with lsmod and the e1000 driver module was not loaded. Confirmed the e1000 driver directory was not present in /sys/bus/pci/drivers

Tried modprobe e1000 but this came back with the following error
modprobe: ERROR: could not insert 'e1000': Key was rejected by service

So out of 7 or 8 reboots the vm booted with the network drivers loaded once.

Regards
Adolf
> 
>> On 11 May 2022, at 20:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
>>
>> Hi All,
>>
>> On 11/05/2022 20:48, Jon Murphy wrote:
>>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
>>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c*
>>> Shouldn’t this be Core Update 168?
>> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
>>
>> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
>>
>> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
>>
>> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
>>
>> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.
> 
> That seems to be a bug that should not be there. Did we recently update apache?
> 
> -Michael
> 
>>
>> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
>>
>> I think I am going to raise a bug on this.
>>
>> Regards,
>> Adolf.
>>> Jon
>>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka(a)ipfire.org <mailto:adolf.belka(a)ipfire.org>> wrote:
>>>>
>>>> Hi All,
>>>>
>>>> On 11/05/2022 16:00, Adolf Belka wrote:
>>>>> Hi Leo,
>>>>>
>>>>> On 11/05/2022 15:26, Leo Hofmann wrote:
>>>>>> Hi Adolf,
>>>>>>
>>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
>>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 <https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762>
>>>>> Now I see it in the code. Thanks very much.
>>>>>>
>>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
>>>>> Thanks for your help anyway. It's got rid of one of my questions.
>>>>>
>>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
>>>>> Will wait to hear other inputs on this problem.
>>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
>>>>>
>>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
>>>>
>>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
>>>>
>>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
>>>>
>>>> Regards,
>>>>
>>>> Adolf.
>>>>
>>>>> Regards,
>>>>> Adolf.
>>>>>
>>>>>>
>>>>>> Best regards
>>>>>> Leo
>>>>>>
>>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>>>>>>
>>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>>>>>>
>>>>>>>
>>>>>>> First try:-
>>>>>>>
>>>>>>> After running the update, which went quite quickly, the bottom of the screen had
>>>>>>>
>>>>>>> Core Update 167 Development Build: master/c22d834c
>>>>>>>
>>>>>>> I couldn't remember if this is what is expected or not for Testing releases.
>>>>>>>
>>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>>>>>>
>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>>
>>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>>>>>>
>>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>>>>>>
>>>>>>> starting
>>>>>>> DUID 00:04:70:........
>>>>>>> red0: interface not found
>>>>>>>
>>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>>>>>>
>>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same
>>>>>>>
>>>>>>> Loading of module with unavailable key is rejected
>>>>>>>
>>>>>>>
>>>>>>> Second try:-
>>>>>>>
>>>>>>> This time running the update took over 10 mins.
>>>>>>>
>>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>>>>>>
>>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>>>>>>
>>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>>>>>>
>>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>>>>>>
>>>>>>> Repository was set back to Stable.
>>>>>>>
>>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>>>>>>
>>>>>>> Before running the pakfire directory had the following
>>>>>>>
>>>>>>> -rw-r--r--  1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>> -rw-r--r--  1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>
>>>>>>> After running it had
>>>>>>>
>>>>>>> -rw-r--r--  1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>>> -rw-r--r--  1 root root 604K May 11 13:24 update-core-upgrade-167.log
>>>>>>> -rw-r--r--  1 root root  175 May 11 13:24 update-core-upgrade-168.log
>>>>>>>
>>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>>>>>>
>>>>>>> Rebooted.
>>>>>>>
>>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>>>>>>
>>>>>>> /opt/pakfire/db/core/mine has 168 in it
>>>>>>>
>>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>>>>>>
>>>>>>>
>>>>>>> Third try:-
>>>>>>>
>>>>>>> -rw-r--r--  1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>>>
>>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>>>>>>
>>>>>>> -rw-r--r--  1 root root 604K May 11 14:00 update-core-upgrade-167.log
>>>>>>> -rw-r--r--  1 root root  63K May 11 14:01 update-core-upgrade-168.log
>>>>>>>
>>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>>>>>>
>>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>>>>>>
>>>>>>>
>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>>>>>>
>>>>>>> Rebooted
>>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>>> Several repeats of the following two lines on the console screen:-
>>>>>>>
>>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>>>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>>>
>>>>>>>
>>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>>>>>>
>>>>>>>
>>>>>>> I have given up at this point.
>>>>>>>
>>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>>>>>>
>>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Adolf.
>>>>>>>
> 

Re: Feedback on problems with Core Update 168 Testing

[Rob Brewer]

[-- Attachment #1: Type: text/plain, Size: 2158 bytes --]

On Thursday 12 May 2022 10:13 Michael Tremer wrote:

> Hello,
> 
> Thanks for spending so much time on this. We definitely need to improve
> the general update experience since we sometimes seem to break people’s
> systems and it is not nice to re-install a firewall from scratch. It will
> take a while.
> 
> So what I can say is that the kernel module issues come from when the
> running kernel is changed and the kernel is trying to load any modules
> that now have changed. This fails by design, because we sign our kernel
> modules. The key is randomly generated at build time and used to sign all
> modules and it then thrown away. For each build, we are using a different,
> unique key that is not preserved.
> 
> This means that although the kernel modules are of the same version, they
> cannot be loaded because the signature check fails. That might also
> explain why you are seeing so many ipset errors, because the kernel cannot
> load that module any more. However, we use so much ipset now, why isn’t
> the module loaded from before the update was started?
> 
> The same goes for any network drivers. I assume you are using virtio or a
> generic e1000 network adapter which will have been initialised at boot
> time. The kernel should never unload the kernel module for that interface
> and load it again later. I have no idea what could have triggered that.
> 
> No matter what though; after you reboot, the new kernel should be booted
> being able to load all modules it wants and the system should run
> absolutely fine. Can you confirm that that is at least the case?
> 



My Pakfire upgrade to 168 on my development APU2 board failed during upgrade 
and I lost ethernet communication with the PC.

The APU2 now fails after the grub prompt with the error: 

*IPFire 2.27 (x86_64) - core166 Development Build: master/8f696f60 GNU/Linu

Loading Linux 5.15.23-ipfire ...
error: file `/vmlinuz-5.15.23-ipfire' not found.
Loading initial ramdisk ...
error: you need to load the kernel first.

so it looks like update-initramfs didn't run after the upgrade.

I'll try to boot the box from a usbstick and see if I can access the disk.

Rob

Re: Feedback on problems with Core Update 168 Testing

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 16112 bytes --]

Hello,

Thanks for spending so much time on this. We definitely need to improve the general update experience since we sometimes seem to break people’s systems and it is not nice to re-install a firewall from scratch. It will take a while.

So what I can say is that the kernel module issues come from when the running kernel is changed and the kernel is trying to load any modules that now have changed. This fails by design, because we sign our kernel modules. The key is randomly generated at build time and used to sign all modules and it then thrown away. For each build, we are using a different, unique key that is not preserved.

This means that although the kernel modules are of the same version, they cannot be loaded because the signature check fails. That might also explain why you are seeing so many ipset errors, because the kernel cannot load that module any more. However, we use so much ipset now, why isn’t the module loaded from before the update was started?

The same goes for any network drivers. I assume you are using virtio or a generic e1000 network adapter which will have been initialised at boot time. The kernel should never unload the kernel module for that interface and load it again later. I have no idea what could have triggered that.

No matter what though; after you reboot, the new kernel should be booted being able to load all modules it wants and the system should run absolutely fine. Can you confirm that that is at least the case?

> On 11 May 2022, at 20:08, Adolf Belka <adolf.belka(a)ipfire.org> wrote:
> 
> Hi All,
> 
> On 11/05/2022 20:48, Jon Murphy wrote:
>> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
>> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c*
>> Shouldn’t this be Core Update 168?
> That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.
> 
> I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.
> 
> I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.
> 
> I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.
> 
> Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.

That seems to be a bug that should not be there. Did we recently update apache?

-Michael

> 
> Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.
> 
> I think I am going to raise a bug on this.
> 
> Regards,
> Adolf.
>> Jon
>>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka(a)ipfire.org <mailto:adolf.belka(a)ipfire.org>> wrote:
>>> 
>>> Hi All,
>>> 
>>> On 11/05/2022 16:00, Adolf Belka wrote:
>>>> Hi Leo,
>>>> 
>>>> On 11/05/2022 15:26, Leo Hofmann wrote:
>>>>> Hi Adolf,
>>>>> 
>>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
>>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 <https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762>
>>>> Now I see it in the code. Thanks very much.
>>>>> 
>>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
>>>> Thanks for your help anyway. It's got rid of one of my questions.
>>>> 
>>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
>>>> Will wait to hear other inputs on this problem.
>>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
>>>> 
>>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
>>> 
>>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
>>> 
>>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
>>> 
>>> Regards,
>>> 
>>> Adolf.
>>> 
>>>> Regards,
>>>> Adolf.
>>>> 
>>>>> 
>>>>> Best regards
>>>>> Leo
>>>>> 
>>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>>>>>> Hi All,
>>>>>> 
>>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>>>>> 
>>>>>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>>>>> 
>>>>>> 
>>>>>> First try:-
>>>>>> 
>>>>>> After running the update, which went quite quickly, the bottom of the screen had
>>>>>> 
>>>>>> Core Update 167 Development Build: master/c22d834c
>>>>>> 
>>>>>> I couldn't remember if this is what is expected or not for Testing releases.
>>>>>> 
>>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>>>>> 
>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>> 
>>>>>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>>>>> 
>>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>>>>> 
>>>>>> starting
>>>>>> DUID 00:04:70:........
>>>>>> red0: interface not found
>>>>>> 
>>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>>>>> 
>>>>>> Looked in bootlog and this was filled with around 4000 lines all the same
>>>>>> 
>>>>>> Loading of module with unavailable key is rejected
>>>>>> 
>>>>>> 
>>>>>> Second try:-
>>>>>> 
>>>>>> This time running the update took over 10 mins.
>>>>>> 
>>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>>>>> 
>>>>>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>>>>> 
>>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>>>>> 
>>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>>>>> 
>>>>>> Repository was set back to Stable.
>>>>>> 
>>>>>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>>>>> 
>>>>>> Before running the pakfire directory had the following
>>>>>> 
>>>>>> -rw-r--r--  1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>> -rw-r--r--  1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>> 
>>>>>> After running it had
>>>>>> 
>>>>>> -rw-r--r--  1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>>> -rw-r--r--  1 root root 604K May 11 13:24 update-core-upgrade-167.log
>>>>>> -rw-r--r--  1 root root  175 May 11 13:24 update-core-upgrade-168.log
>>>>>> 
>>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>>>>> 
>>>>>> Rebooted.
>>>>>> 
>>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>>>>> 
>>>>>> /opt/pakfire/db/core/mine has 168 in it
>>>>>> 
>>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>>>>> 
>>>>>> 
>>>>>> Third try:-
>>>>>> 
>>>>>> -rw-r--r--  1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>> 
>>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>>>>> 
>>>>>> -rw-r--r--  1 root root 604K May 11 14:00 update-core-upgrade-167.log
>>>>>> -rw-r--r--  1 root root  63K May 11 14:01 update-core-upgrade-168.log
>>>>>> 
>>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>>>>> 
>>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>>>>> 
>>>>>> 
>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>>>>> 
>>>>>> Rebooted
>>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>> Several repeats of the following two lines on the console screen:-
>>>>>> 
>>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>> 
>>>>>> 
>>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>>>>> 
>>>>>> 
>>>>>> I have given up at this point.
>>>>>> 
>>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>>>>> 
>>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>>>>> 
>>>>>> Regards,
>>>>>> 
>>>>>> Adolf.
>>>>>> 

Re: Feedback on problems with Core Update 168 Testing

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 14156 bytes --]

Hi All,

On 11/05/2022 20:48, Jon Murphy wrote:
> I just did an update from CU 167 (stable) to CU 168 (testing) and I got the same build info:
> 
> *IPFire 2.27 (x86_64) - Core Update 167 Development Build: master/c22d834c*
> 
> Shouldn’t this be Core Update 168?
That was what I thought but I couldn't precisely remember what it was in the past with Testing Releases.

I then downloaded the Core 168 iso from the nightlies build of master/latest/x86_64 and did an install on the First attempt vm.

I got to the stage of entering the root and admin passwords. Entered what I usually do for the vm's and after pressing OK on the admin password screen I got as message box saying "Problem setting IPFire 'admin' user password". I pressed the OK button and got another message box saying "Initial setup was not entirely complete. You must ensure that Setup is properly finished by running setup again at the shell.". Pressing the OK button on that message box causes setup to restart but again after entering a valid admin password twice I got the same "Problem setting IPFire 'admin' user password" message box.

I then tried installing CU168 from the same iso onto a clone of my running CU167 vm. Same result with "Problem setting IPFire 'admin' user password" message box.

Then I created a completely new vm from scratch and tried installing CU168 Testing iso and again got "Problem setting IPFire 'admin' user password" message box.

Don't know if this is related to the problem with the missing interfaces but if not then it is another problem.

I think I am going to raise a bug on this.

Regards,
Adolf.
> 
> Jon
> 
> 
>> On May 11, 2022, at 9:19 AM, Adolf Belka <adolf.belka(a)ipfire.org <mailto:adolf.belka(a)ipfire.org>> wrote:
>>
>> Hi All,
>>
>> On 11/05/2022 16:00, Adolf Belka wrote:
>>> Hi Leo,
>>>
>>> On 11/05/2022 15:26, Leo Hofmann wrote:
>>>> Hi Adolf,
>>>>
>>>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
>>> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762 <https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762>
>>> Now I see it in the code. Thanks very much.
>>>>
>>>> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
>>> Thanks for your help anyway. It's got rid of one of my questions.
>>>
>>> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
>>> Will wait to hear other inputs on this problem.
>>> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
>>>
>> Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.
>>
>> I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.
>>
>> When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/
>>
>> Regards,
>>
>> Adolf.
>>
>>> Regards,
>>> Adolf.
>>>
>>>>
>>>> Best regards
>>>> Leo
>>>>
>>>> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>>>>> Hi All,
>>>>>
>>>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>>>>
>>>>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>>>>
>>>>>
>>>>> First try:-
>>>>>
>>>>> After running the update, which went quite quickly, the bottom of the screen had
>>>>>
>>>>> Core Update 167 Development Build: master/c22d834c
>>>>>
>>>>> I couldn't remember if this is what is expected or not for Testing releases.
>>>>>
>>>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>>>>
>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>>
>>>>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>>>>
>>>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>>>>
>>>>> starting
>>>>> DUID 00:04:70:........
>>>>> red0: interface not found
>>>>>
>>>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>>>>
>>>>> Looked in bootlog and this was filled with around 4000 lines all the same
>>>>>
>>>>> Loading of module with unavailable key is rejected
>>>>>
>>>>>
>>>>> Second try:-
>>>>>
>>>>> This time running the update took over 10 mins.
>>>>>
>>>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>>>>
>>>>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>>>>
>>>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>>>>
>>>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>>>>
>>>>> Repository was set back to Stable.
>>>>>
>>>>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>>>>
>>>>> Before running the pakfire directory had the following
>>>>>
>>>>> -rw-r--r--  1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>> -rw-r--r--  1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>
>>>>> After running it had
>>>>>
>>>>> -rw-r--r--  1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>>>> -rw-r--r--  1 root root 604K May 11 13:24 update-core-upgrade-167.log
>>>>> -rw-r--r--  1 root root  175 May 11 13:24 update-core-upgrade-168.log
>>>>>
>>>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>>>>
>>>>> Rebooted.
>>>>>
>>>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>>>>
>>>>> /opt/pakfire/db/core/mine has 168 in it
>>>>>
>>>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>>>>
>>>>>
>>>>> Third try:-
>>>>>
>>>>> -rw-r--r--  1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>>>
>>>>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>>>>
>>>>> -rw-r--r--  1 root root 604K May 11 14:00 update-core-upgrade-167.log
>>>>> -rw-r--r--  1 root root  63K May 11 14:01 update-core-upgrade-168.log
>>>>>
>>>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>>>>
>>>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>>>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>>>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>>>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>>>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>>>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com <http://ipfire.earl-net.com/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>>>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>>>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>>>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>>>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>>>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>>>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>>>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>>>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>>>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>>>>
>>>>>
>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>>>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>>>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org <http://mirror1.ipfire.org/> (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>>>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>>>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>>>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>>>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>>>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>>>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>>>>
>>>>> Rebooted
>>>>> Lots of ipset restore error but went by too fast to get more detail.
>>>>> Several repeats of the following two lines on the console screen:-
>>>>>
>>>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>>>>> Perhaps iptables or your kernel needs to be upgraded.
>>>>>
>>>>>
>>>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>>>>
>>>>>
>>>>> I have given up at this point.
>>>>>
>>>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>>>>
>>>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Adolf.
>>>>>
> 

Re: Feedback on problems with Core Update 168 Testing

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 11698 bytes --]

Hi All,

On 11/05/2022 16:00, Adolf Belka wrote:
> Hi Leo,
>
> On 11/05/2022 15:26, Leo Hofmann wrote:
>> Hi Adolf,
>>
>> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
> So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
>> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762
> Now I see it in the code. Thanks very much.
>>
>> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
> Thanks for your help anyway. It's got rid of one of my questions.
>
> Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
> Will wait to hear other inputs on this problem.
> Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.
>
Running setup does not help. None of the interfaces are available to be assigned to the red, green etc zones.

I checked the interfaces in the vm setup and they are as previously defined and previously working on all earlier CU's of my vm.

When rebooting I did see a fast scrolling message that seemed to say something like Invalid Kernel Argument but that is as much as I was able to see/

Regards,

Adolf.

> Regards,
> Adolf.
>
>>
>> Best regards
>> Leo
>>
>> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>>> Hi All,
>>>
>>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>>
>>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>>
>>>
>>> First try:-
>>>
>>> After running the update, which went quite quickly, the bottom of the screen had
>>>
>>> Core Update 167 Development Build: master/c22d834c
>>>
>>> I couldn't remember if this is what is expected or not for Testing releases.
>>>
>>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>>
>>> Lots of ipset restore error but went by too fast to get more detail.
>>>
>>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>>
>>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>>
>>> starting
>>> DUID 00:04:70:........
>>> red0: interface not found
>>>
>>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>>
>>> Looked in bootlog and this was filled with around 4000 lines all the same
>>>
>>> Loading of module with unavailable key is rejected
>>>
>>>
>>> Second try:-
>>>
>>> This time running the update took over 10 mins.
>>>
>>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>>
>>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>>
>>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>>
>>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>>
>>> Repository was set back to Stable.
>>>
>>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>>
>>> Before running the pakfire directory had the following
>>>
>>> -rw-r--r--  1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>> -rw-r--r--  1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>
>>> After running it had
>>>
>>> -rw-r--r--  1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>>> -rw-r--r--  1 root root 604K May 11 13:24 update-core-upgrade-167.log
>>> -rw-r--r--  1 root root  175 May 11 13:24 update-core-upgrade-168.log
>>>
>>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>>
>>> Rebooted.
>>>
>>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>>
>>> /opt/pakfire/db/core/mine has 168 in it
>>>
>>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>>
>>>
>>> Third try:-
>>>
>>> -rw-r--r--  1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>>
>>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>>
>>> -rw-r--r--  1 root root 604K May 11 14:00 update-core-upgrade-167.log
>>> -rw-r--r--  1 root root  63K May 11 14:01 update-core-upgrade-168.log
>>>
>>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>>
>>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>>
>>>
>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>>
>>> Rebooted
>>> Lots of ipset restore error but went by too fast to get more detail.
>>> Several repeats of the following two lines on the console screen:-
>>>
>>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>>> Perhaps iptables or your kernel needs to be upgraded.
>>>
>>>
>>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>>
>>>
>>> I have given up at this point.
>>>
>>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>>
>>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>>
>>> Regards,
>>>
>>> Adolf.
>>>

Re: Feedback on problems with Core Update 168 Testing

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 11028 bytes --]

Hi Leo,

On 11/05/2022 15:26, Leo Hofmann wrote:
> Hi Adolf,
> 
> Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
So that proves that I haven't been looking so closely on my upgrades in the past because I have never noticed that.
> https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762
Now I see it in the code. Thanks very much.
> 
> But unfortunately this is all I can contribute, my test system updated to 168 without problems!
Thanks for your help anyway. It's got rid of one of my questions.

Just need to understand now why 2 out of 3 updates resulted in a complete loss of the interface assignments.
Will wait to hear other inputs on this problem.
Maybe will try running setup on the first try I had to confirm that I can re-assign the interfaces again.

Regards,
Adolf.

> 
> Best regards
> Leo
> 
> Am 11.05.2022 um 15:11 schrieb Adolf Belka:
>> Hi All,
>>
>> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>>
>> Normally my updates go without any real problems. This is the first time where this is not the case.
>>
>>
>> First try:-
>>
>> After running the update, which went quite quickly, the bottom of the screen had
>>
>> Core Update 167 Development Build: master/c22d834c
>>
>> I couldn't remember if this is what is expected or not for Testing releases.
>>
>> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>>
>> Lots of ipset restore error but went by too fast to get more detail.
>>
>> Tried to access via the WUI and no access. Tried via SSH and no access.
>>
>> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>>
>> starting
>> DUID 00:04:70:........
>> red0: interface not found
>>
>> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>>
>> Looked in bootlog and this was filled with around 4000 lines all the same
>>
>> Loading of module with unavailable key is rejected
>>
>>
>> Second try:-
>>
>> This time running the update took over 10 mins.
>>
>> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>>
>> /opt/pakfire/db/core/mine contained 167 beforehand.
>>
>> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>>
>> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>>
>> Repository was set back to Stable.
>>
>> Changed again to Testing and it said again that there was an update from 167 to 168.
>>
>> Before running the pakfire directory had the following
>>
>> -rw-r--r--  1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>> -rw-r--r--  1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>
>> After running it had
>>
>> -rw-r--r--  1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
>> -rw-r--r--  1 root root 604K May 11 13:24 update-core-upgrade-167.log
>> -rw-r--r--  1 root root  175 May 11 13:24 update-core-upgrade-168.log
>>
>> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>>
>> Rebooted.
>>
>> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>>
>> /opt/pakfire/db/core/mine has 168 in it
>>
>> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>>
>>
>> Third try:-
>>
>> -rw-r--r--  1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>>
>> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>>
>> -rw-r--r--  1 root root 604K May 11 14:00 update-core-upgrade-167.log
>> -rw-r--r--  1 root root  63K May 11 14:01 update-core-upgrade-168.log
>>
>> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>>
>> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
>> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
>> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
>> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
>> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
>> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
>> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
>> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
>> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
>> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
>> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
>> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
>> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
>> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
>> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
>> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
>> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
>> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
>> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
>> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
>> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
>> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>>
>>
>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
>> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
>> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
>> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
>> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
>> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
>> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
>> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
>> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
>> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
>> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
>> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
>> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
>> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>>
>> Rebooted
>> Lots of ipset restore error but went by too fast to get more detail.
>> Several repeats of the following two lines on the console screen:-
>>
>> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
>> Perhaps iptables or your kernel needs to be upgraded.
>>
>>
>> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>>
>>
>> I have given up at this point.
>>
>> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>>
>> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>>
>> Regards,
>>
>> Adolf.
>>

Re: Feedback on problems with Core Update 168 Testing

[Leo Hofmann]

[-- Attachment #1: Type: text/plain, Size: 10268 bytes --]

Hi Adolf,

Pakfire always automatically reinstalls the current release before updating if you are in the testing branch. See this function in the Pakfire code:
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/pakfire/lib/functions.pl;h=d4e338f23ae8ae97d6f18c6d8890d13463dc5d30;hb=refs/heads/next#l762

But unfortunately this is all I can contribute, my test system updated to 168 without problems!

Best regards
Leo

Am 11.05.2022 um 15:11 schrieb Adolf Belka:
> Hi All,
>
> I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.
>
> Normally my updates go without any real problems. This is the first time where this is not the case.
>
>
> First try:-
>
> After running the update, which went quite quickly, the bottom of the screen had
>
> Core Update 167 Development Build: master/c22d834c
>
> I couldn't remember if this is what is expected or not for Testing releases.
>
> Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.
>
> Lots of ipset restore error but went by too fast to get more detail.
>
> Tried to access via the WUI and no access. Tried via SSH and no access.
>
> Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message
>
> starting
> DUID 00:04:70:........
> red0: interface not found
>
> So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.
>
> Looked in bootlog and this was filled with around 4000 lines all the same
>
> Loading of module with unavailable key is rejected
>
>
> Second try:-
>
> This time running the update took over 10 mins.
>
> Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.
>
> /opt/pakfire/db/core/mine contained 167 beforehand.
>
> Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.
>
> Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28
>
> Repository was set back to Stable.
>
> Changed again to Testing and it said again that there was an update from 167 to 168.
>
> Before running the pakfire directory had the following
>
> -rw-r--r--  1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
> -rw-r--r--  1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>
> After running it had
>
> -rw-r--r--  1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
> -rw-r--r--  1 root root 604K May 11 13:24 update-core-upgrade-167.log
> -rw-r--r--  1 root root  175 May 11 13:24 update-core-upgrade-168.log
>
> back to Core Update 167 Development Build: master/c22d834c at bottom of screen.
>
> Rebooted.
>
> All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c
>
> /opt/pakfire/db/core/mine has 168 in it
>
> I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.
>
>
> Third try:-
>
> -rw-r--r--  1 root root 323K Apr 28 09:47 update-core-upgrade-167.log
>
> Selected Testing and ran update from 167 to 168 which occurred very quickly.
>
> -rw-r--r--  1 root root 604K May 11 14:00 update-core-upgrade-167.log
> -rw-r--r--  1 root root  63K May 11 14:01 update-core-upgrade-168.log
>
> Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.
>
> May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
> May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
> May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
> May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
> May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
> May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
> May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
> May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
> May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
> May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
> May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
> May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
> May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
> May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
> May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
> May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
> May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
> May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
> May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
> May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
> May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
> May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
> May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
> May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
> May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
> May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...
>
>
> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
> May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
> May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
> May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
> May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
> May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
> May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
> May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
> May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
> May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
> May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
> May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
> May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
> May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.
>
> Rebooted
> Lots of ipset restore error but went by too fast to get more detail.
> Several repeats of the following two lines on the console screen:-
>
> iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
> Perhaps iptables or your kernel needs to be upgraded.
>
>
> Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.
>
>
> I have given up at this point.
>
> I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.
>
> If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.
>
> Regards,
>
> Adolf.
>

Feedback on problems with Core Update 168 Testing

[Adolf Belka]

[-- Attachment #1: Type: text/plain, Size: 9510 bytes --]

Hi All,

I have tried to update my Core Update 167 vm machine three times now (using a clone) and have had several problems so I thought I would outline what has happened for further thoughts.

Normally my updates go without any real problems. This is the first time where this is not the case.


First try:-

After running the update, which went quite quickly, the bottom of the screen had

Core Update 167 Development Build: master/c22d834c

I couldn't remember if this is what is expected or not for Testing releases.

Rebooted the vm and on booting a large number of error messages scrolled across the console screen. My Virtualbox terminal for IPFire has no scroll capability so I can only write on things I saw as they flew past or what was on the screen when it stopped.

Lots of ipset restore error but went by too fast to get more detail.

Tried to access via the WUI and no access. Tried via SSH and no access.

Checked the red0 directory and there were no files present. Tried restarting red0 and got the following message

starting
DUID 00:04:70:........
red0: interface not found

So I ran ip address show and it came up with only the lo interface. No red0, green0, blue0 or orange0.

Looked in bootlog and this was filled with around 4000 lines all the same

Loading of module with unavailable key is rejected


Second try:-

This time running the update took over 10 mins.

Noticed on the pakfire screen that it first ran a Core Update 167 upgrade then it ran the Core Update 168. My IPFire vm was definitely on Core Update 167.

/opt/pakfire/db/core/mine contained 167 beforehand.

Before doing the reboot I accessed via ssh and saw that in the /var/log/pakfire/ directory the update-core-upgrade-167 file has been updated as well as there being a 168. Both files had the same last modified date/time.

Rebooted. This time everything came back up okay but the bottom of the screen now just said Core Update 167 and there was no update-core-upgrade-168 file in the pakfire directory and the 167 version was back to its date of Apr 28

Repository was set back to Stable.

Changed again to Testing and it said again that there was an update from 167 to 168.

Before running the pakfire directory had the following

-rw-r--r--  1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
-rw-r--r--  1 root root 323K Apr 28 09:47 update-core-upgrade-167.log

After running it had

-rw-r--r--  1 root root 1.8K Mar 31 23:26 update-core-upgrade-166.log
-rw-r--r--  1 root root 604K May 11 13:24 update-core-upgrade-167.log
-rw-r--r--  1 root root  175 May 11 13:24 update-core-upgrade-168.log

back to Core Update 167 Development Build: master/c22d834c at bottom of screen.

Rebooted.

All interfaces present. Pakfire under System Status: says Core-Update-Level: 168 and the bottom of the screen showed Core Update 167 Development Build: master/c22d834c

/opt/pakfire/db/core/mine has 168 in it

I checked the WIO wio.pl file that had a bug fix and it was as expected for CU168.


Third try:-

-rw-r--r--  1 root root 323K Apr 28 09:47 update-core-upgrade-167.log

Selected Testing and ran update from 167 to 168 which occurred very quickly.

-rw-r--r--  1 root root 604K May 11 14:00 update-core-upgrade-167.log
-rw-r--r--  1 root root  63K May 11 14:01 update-core-upgrade-168.log

Here is the log file info for pakfire from messages.1.gz and messages which shows it downloading and upgrading CU167 first then doing CU168 as far as I can tell.

May 11 13:59:30 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27.1-x86_64 started!
May 11 13:59:30 ipfire pakfire: CORE INFO: core-list.db is 27 seconds old. - DEBUG: noforce
May 11 13:59:30 ipfire pakfire: CORE UPGR: Upgrading from release 166 to 168
May 11 13:59:30 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-167.ipfire
May 11 13:59:30 ipfire pakfire: MIRROR INFO: 2 servers found in list
May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
May 11 13:59:30 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire has size of 71914825 bytes
May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-167.ipfire is fine.
May 11 13:59:32 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-167.ipfire
May 11 13:59:32 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-168
May 11 13:59:32 ipfire pakfire: MIRROR INFO: 2 servers found in list
May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org (HTTPS) - File: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
May 11 13:59:32 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168 has size of 1804 bytes
May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-168 is fine.
May 11 13:59:33 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/meta/meta-core-upgrade-168
May 11 13:59:33 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-168.ipfire
May 11 13:59:33 ipfire pakfire: MIRROR INFO: 2 servers found in list
May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.earl-net.com (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
May 11 13:59:33 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire has size of 34355484 bytes
May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
May 11 13:59:34 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-168.ipfire is fine.
May 11 13:59:34 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/core-upgrade-2.27-168.ipfire
May 11 13:59:34 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Decrypting...
May 11 13:59:34 ipfire pakfire: CLEANUP: tmp
May 11 13:59:34 ipfire pakfire: DECRYPT STARTED: core-upgrade-167
May 11 13:59:35 ipfire pakfire: DECRYPT FINISHED: core-upgrade-167 - Status: 0
May 11 13:59:35 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Upgrading files and running post-upgrading scripts...
May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-167: Finished.
May 11 14:00:31 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Decrypting...
May 11 14:00:31 ipfire pakfire: CLEANUP: tmp
May 11 14:00:31 ipfire pakfire: DECRYPT STARTED: core-upgrade-168
May 11 14:00:32 ipfire pakfire: DECRYPT FINISHED: core-upgrade-168 - Status: 0
May 11 14:00:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Upgrading files and running post-upgrading scripts...


May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: core-upgrade-168: Finished.
May 11 14:01:10 ipfire pakfire: CORE INFO: core-list.db is 127 seconds old. - DEBUG: noforce
May 11 14:01:10 ipfire pakfire: PAKFIRE RESV: wio: Resolving dependencies...
May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: We are going to install all packages listed above.
May 11 14:01:10 ipfire pakfire: DOWNLOAD STARTED: paks/wio-1.3.2-15.ipfire
May 11 14:01:10 ipfire pakfire: MIRROR INFO: 2 servers found in list
May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Host: mirror1.ipfire.org (HTTPS) - File: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire has size of 47216 bytes
May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
May 11 14:01:10 ipfire pakfire: DOWNLOAD INFO: Signature of wio-1.3.2-15.ipfire is fine.
May 11 14:01:10 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27.1-x86_64/paks/wio-1.3.2-15.ipfire
May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Decrypting...
May 11 14:01:10 ipfire pakfire: CLEANUP: tmp
May 11 14:01:10 ipfire pakfire: DECRYPT STARTED: wio
May 11 14:01:10 ipfire pakfire: DECRYPT FINISHED: wio - Status: 0
May 11 14:01:10 ipfire pakfire: PAKFIRE UPGR: wio: Upgrading files and running post-upgrading scripts...
May 11 14:01:11 ipfire pakfire: CLEANUP: tmp
May 11 14:01:11 ipfire pakfire: PAKFIRE UPGR: wio: Finished.
May 11 14:01:11 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.

Rebooted
Lots of ipset restore error but went by too fast to get more detail.
Several repeats of the following two lines on the console screen:-

iptables v1.8.7 (legacy): can't initialize iptables table `filter' : Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.


Only lo interface present. All other interfaces red0, green0, blue0, orange0 not present.


I have given up at this point.

I have kept the clones of all three attempts so if you need any additional info from logs or files just let me know.

If this is something obviously wrong that I am doing, please let me know. If not then I will probably raise a bug on this.

Regards,

Adolf.