* Core 159 - "Kernel errors present" in LOG SUMMARY @ 2021-08-27 16:17 Matthias Fischer 0 siblings, 0 replies; 8+ messages in thread From: Matthias Fischer @ 2021-08-27 16:17 UTC (permalink / raw) To: development [-- Attachment #1: Type: text/plain, Size: 122 bytes --] Hi, I forgot: Profile is: https://fireinfo.ipfire.org/profile/5f68a6360ffbecb6877dcac75f5b8c8030f43ce8 Best, Matthias ^ permalink raw reply [flat|nested] 8+ messages in thread
[parent not found: <F92EEFDA-65C9-45D6-8BCE-7ED5B6793313@ipfire.org>]
* Re: Core 159 - "Kernel errors present" in LOG SUMMARY [not found] <F92EEFDA-65C9-45D6-8BCE-7ED5B6793313@ipfire.org> @ 2021-09-09 20:36 ` Peter Müller 2021-09-20 15:16 ` Matthias Fischer 0 siblings, 1 reply; 8+ messages in thread From: Peter Müller @ 2021-09-09 20:36 UTC (permalink / raw) To: development [-- Attachment #1: Type: text/plain, Size: 6281 bytes --] Hello *, just for the records: I noticed this behaviour while testing Core Update 140/141 (in February 2020) the first time. Not kept track on this, but it does not seem to cause any harm - at least none I am aware of. But it certainly is not a good thing to see, either... Please refer to https://lists.ipfire.org/pipermail/development/2020-February/007046.html for further details. Thanks, and best regards, Peter Müller > > >> On 31 Aug 2021, at 16:25, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote: >> >> Hi, >> >> On 31.08.2021 11:56, Michael Tremer wrote: >>> Hey, >>> >>> This is an oops in the code that injects packets back into the kernel after they have been processed by suricata. >> >> Wow. How did you find this out!? > > There are two key functions in the trace: > >>>> 1 Time(s): nf_queue_entry_release_refs+0x82/0xa0 >>>> 1 Time(s): nf_reinject+0x7a/0x1e0 > > They are from Netfilter and the NFQUEUE module. We only use that for the IPS. > >>> Was this a one-off or does this happen on a regular basis? >> >> Until now, it only happened once. >> >> It rebooted the machine - just to be sure - and its running "without >> seen problems" since then. Absolutely normal. > > Would be interesting to see how it behaves if it doesn’t get a reboot. > > This is definitely a bug and needs to be fixed in the Linux kernel. > > -Michael > >> >> Best, >> Matthias >> >>> -Michael >>> >>>> On 27 Aug 2021, at 17:16, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote: >>>> >>>> Hi, >>>> >>>> today I took the usual look in LOG SUMMARY and was surprised finding this: >>>> >>>> ***SNIP*** >>>> Kernel >>>> >>>> WARNING: Kernel Errors Present >>>> WARNING: CPU: 0 PID: 2984 at lib/refcount.c:28 r ...: 1 Time(s) >>>> >>>> 1 Time(s): ? nfnetlink_net_exit_batch+0x60/0x60 >>>> 1 Time(s): [last unloaded: hwmon_vid] >>>> 1 Time(s): ____sys_sendmsg+0x258/0x2a0 >>>> 1 Time(s): ___sys_sendmsg+0xa3/0xf0 >>>> 1 Time(s): __sys_sendmsg+0x81/0xd0 >>>> 1 Time(s): do_syscall_64+0x33/0x40 >>>> 1 Time(s): entry_SYSCALL_64_after_hwframe+0x44/0xa9 >>>> 1 Time(s): netlink_rcv_skb+0x5b/0x100 >>>> 1 Time(s): netlink_sendmsg+0x23a/0x470 >>>> 1 Time(s): netlink_unicast+0x209/0x2d0 >>>> 1 Time(s): nf_queue_entry_release_refs+0x82/0xa0 >>>> 1 Time(s): nf_reinject+0x7a/0x1e0 >>>> 1 Time(s): nfnetlink_rcv_msg+0x16d/0x2c0 >>>> 1 Time(s): nfqnl_recv_verdict+0x302/0x4f0 [nfnetlink_queue] >>>> 1 Time(s): sock_sendmsg+0x5e/0x60 >>>> 1 Time(s): ------------[ cut here ]------------ >>>> 1 Time(s): ---[ end trace 49e1e291edb98731 ]--- >>>> 1 Time(s): CPU: 0 PID: 2984 Comm: W-NFQ#1 Tainted: G O >>>> 5.10.55-ipfire #1 >>>> 1 Time(s): CR2: 000079fa1356a6f8 CR3: 0000000101532004 CR4: >>>> 00000000000706f0 >>>> 1 Time(s): CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>> 1 Time(s): Call Trace: >>>> 1 Time(s): Code: 05 ba 60 1f 01 01 e8 af a3 52 00 0f 0b c3 80 3d a8 60 >>>> 1f 01 00 75 95 48 c7 c7 b8 fc 11 b4 c6 05 98 60 1f 01 01 e8 90 a3 52 00 >>>> <0f> 0b c3 80 3d 87 60 1f 01 00 0f 85 72 ff ff ff 48 c7 c7 10 fd 11 >>>> 1 Time(s): Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 fa ee ff >>>> ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2e 00 00 00 0f 05 >>>> <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 4e ef ff ff 48 >>>> 1 Time(s): FS: 00007144d27d9640(0000) GS:ffff9db887200000(0000) >>>> knlGS:0000000000000000 >>>> 1 Time(s): Hardware name: To be filled by O.E.M. To be filled by >>>> O.E.M./CRESCENTBAY, BIOS 5.011 04/13/2016 >>>> 1 Time(s): Modules linked in: xt_REDIRECT nfnetlink_queue xt_NFQUEUE >>>> xt_MASQUERADE ccm cpufreq_conservative cpufreq_ondemand xt_geoip(O) >>>> xt_ipp2p(O) compat_xtables(O) xt_hashlimit xt_mac xt_multiport xt_mark >>>> xt_policy xt_TCPMSS xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 >>>> xt_LOG xt_limit nf_log_ipv4 nf_log_common iptable_raw iptable_mangle >>>> iptable_filter vfat fat sch_fq_codel rt2800usb rt2x00usb >>>> x86_pkg_temp_thermal rt2800lib intel_powerclamp coretemp rt2x00lib >>>> kvm_intel mac80211 kvm at24 regmap_i2c iTCO_wdt iTCO_vendor_support >>>> irqbypass crct10dif_pclmul cfg80211 crc32_pclmul ghash_clmulni_intel >>>> pcspkr i2c_i801 rfkill r8169 lpc_ich i2c_smbus mfd_core realtek libarc4 >>>> ir_rc6_decoder rc_rc6_mce i2c_algo_bit snd_hda_codec_realtek nuvoton_cir >>>> fb_sys_fops syscopyarea sysfillrect sysimgblt snd_hda_codec_generic >>>> ledtrig_audio i2c_core rc_core snd_hda_intel snd_intel_dspcfg >>>> snd_hda_codec snd_hda_core snd_hwdep acpi_pad snd_pcm snd_timer snd >>>> soundcore lp parport_pc parport video >>>> 1 Time(s): R10: 0000000000000000 R11: 0000000000000293 R12: >>>> 0000000000000000 >>>> 1 Time(s): R10: ffffb330821af780 R11: ffffffffb4533dc8 R12: >>>> ffff9db791371980 >>>> 1 Time(s): R13: 0000000000000000 R14: 0000000000000001 R15: >>>> 0000000000000070 >>>> 1 Time(s): R13: ffff9db7444d9200 R14: 0000000000000005 R15: >>>> ffff9db7f15c6b00 >>>> 1 Time(s): RAX: 0000000000000000 RBX: ffff9db791371980 RCX: >>>> 0000000000000027 >>>> 1 Time(s): RAX: ffffffffffffffda RBX: 00007144c4268dd0 RCX: >>>> 00007144d507062d >>>> 1 Time(s): RBP: 00007144d27d6fe0 R08: 0000000000000000 R09: >>>> 00007144d4754de0 >>>> 1 Time(s): RBP: ffffb330821afa50 R08: 0000000000000000 R09: >>>> ffffb330821af788 >>>> 1 Time(s): RDX: 0000000000000000 RSI: 00007144d27d6f80 RDI: >>>> 0000000000000006 >>>> 1 Time(s): RDX: ffff9db887218968 RSI: 0000000000000001 RDI: >>>> ffff9db887218960 >>>> 1 Time(s): RIP: 0010:refcount_warn_saturate+0xa6/0xf0 >>>> 1 Time(s): RIP: 0033:0x7144d507062d >>>> 1 Time(s): RSP: 0018:ffffb330821af950 EFLAGS: 00010282 >>>> 1 Time(s): RSP: 002b:00007144d27d6f40 EFLAGS: 00000293 ORIG_RAX: >>>> 000000000000002e >>>> 1 Time(s): refcount_t: underflow; use-after-free. >>>> ***SNAP*** >>>> >>>> I don't know exactly how do deal with this - does anyone has an idea >>>> what this means? >>>> >>>> Besides the machine is up and running with Core 159 / 64bit since 6 days >>>> now and if I hadn't looked at the logs I would not have noticed it. >>>> >>>> I didn't reboot or changed anything yet- should I do? >>>> >>>> Best, >>>> Matthias >>> >> > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Core 159 - "Kernel errors present" in LOG SUMMARY 2021-09-09 20:36 ` Peter Müller @ 2021-09-20 15:16 ` Matthias Fischer 2021-09-21 9:36 ` Michael Tremer 0 siblings, 1 reply; 8+ messages in thread From: Matthias Fischer @ 2021-09-20 15:16 UTC (permalink / raw) To: development [-- Attachment #1: Type: text/plain, Size: 10525 bytes --] Hi, today it happened again - at 1:18am: ***SNIP*** ipfire kernel: refcount_t: underflow; use-after-free. ipfire kernel: WARNING: CPU: 1 PID: 30228 at lib/refcount.c:28 refcount_warn_saturate+0xa6/0xf0 ipfire kernel: Modules linked in: xt_REDIRECT nfnetlink_queue xt_NFQUEUE xt_MASQUERADE ccm cpufreq_conservative cpufreq_ondemand xt_geoip(O) xt_ipp2p(O) compat_xtables(O) xt_hashlimit xt_mac xt_multiport xt_mark xt_policy xt_TCPMSS xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 xt_LOG xt_limit nf_log_ipv4 nf_log_common iptable_raw iptable_mangle iptable_filter vfat fat rt2800usb rt2x00usb rt2800lib rt2x00lib sch_fq_codel x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel mac80211 at24 kvm regmap_i2c iTCO_wdt iTCO_vendor_support irqbypass crct10dif_pclmul crc32_pclmul cfg80211 ghash_clmulni_intel i2c_i801 pcspkr i2c_smbus lpc_ich rfkill mfd_core r8169 realtek libarc4 ir_rc6_decoder i2c_algo_bit fb_sys_fops snd_hda_codec_realtek syscopyarea rc_rc6_mce sysfillrect snd_hda_codec_generic sysimgblt nuvoton_cir i2c_core ledtrig_audio rc_core snd_hda_intel snd_intel_dspcfg snd_hda_codec snd_hda_core snd_hwdep acpi_pad snd_pcm snd_timer snd soundcore lp parport_pc parport video ipfire kernel: CPU: 1 PID: 30228 Comm: W-NFQ#0 Tainted: G O 5.10.55-ipfire #1 ipfire kernel: Hardware name: To be filled by O.E.M. To be filled by O.E.M./CRESCENTBAY, BIOS 5.011 04/13/2016 ipfire kernel: RIP: 0010:refcount_warn_saturate+0xa6/0xf0 ipfire kernel: Code: 05 ba 60 1f 01 01 e8 af a3 52 00 0f 0b c3 80 3d a8 60 1f 01 00 75 95 48 c7 c7 b8 fc 11 ab c6 05 98 60 1f 01 01 e8 90 a3 52 00 <0f> 0b c3 80 3d 87 60 1f 01 00 0f 85 72 ff ff ff 48 c7 c7 10 fd 11 ipfire kernel: RSP: 0018:ffffbaba82823950 EFLAGS: 00010282 ipfire kernel: RAX: 0000000000000000 RBX: ffff998e77a25500 RCX: 0000000000000027 ipfire kernel: RDX: ffff998f47318968 RSI: 0000000000000001 RDI: ffff998f47318960 ipfire kernel: RBP: ffffbaba82823a50 R08: 0000000000000000 R09: ffffbaba82823788 ipfire kernel: R10: ffffbaba82823780 R11: ffffffffab533dc8 R12: ffff998e77a25500 ipfire kernel: R13: ffff998e04193a00 R14: 0000000000000005 R15: ffff998e03a6ea00 ipfire kernel: FS: 0000758d85125640(0000) GS:ffff998f47300000(0000) knlGS:0000000000000000 ipfire kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 ipfire kernel: CR2: 00007722aefed0a0 CR3: 000000010351e003 CR4: 00000000000706e0 ipfire kernel: Call Trace: ipfire kernel: nf_queue_entry_release_refs+0x82/0xa0 ipfire kernel: nf_reinject+0x7a/0x1e0 ipfire kernel: nfqnl_recv_verdict+0x302/0x4f0 [nfnetlink_queue] ipfire kernel: nfnetlink_rcv_msg+0x16d/0x2c0 ipfire kernel: ? nfnetlink_net_exit_batch+0x60/0x60 ipfire kernel: netlink_rcv_skb+0x5b/0x100 ipfire kernel: netlink_unicast+0x209/0x2d0 ipfire kernel: netlink_sendmsg+0x23a/0x470 ipfire kernel: sock_sendmsg+0x5e/0x60 ipfire kernel: ____sys_sendmsg+0x258/0x2a0 ipfire kernel: ___sys_sendmsg+0xa3/0xf0 ipfire kernel: __sys_sendmsg+0x81/0xd0 ipfire kernel: do_syscall_64+0x33/0x40 ipfire kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 ipfire kernel: RIP: 0033:0x758d871bb62d ipfire kernel: Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 fa ee ff ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 4e ef ff ff 48 ipfire kernel: RSP: 002b:0000758d85122f40 EFLAGS: 00000293 ORIG_RAX: 000000000000002e ipfire kernel: RAX: ffffffffffffffda RBX: 0000758d80268dd0 RCX: 0000758d871bb62d ipfire kernel: RDX: 0000000000000000 RSI: 0000758d85122f80 RDI: 0000000000000005 ipfire kernel: RBP: 0000758d85122fe0 R08: 0000000000000000 R09: 0000758d8689fde0 ipfire kernel: R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000000 ipfire kernel: R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000070 ipfire kernel: ---[ end trace 4c8c047c62e118e2 ]--- ***SNAP*** Machine is running without (seen) problems - I didn'T reboot yet. Best, Matthias On 09.09.2021 22:36, Peter Müller wrote: > Hello *, > > just for the records: I noticed this behaviour while testing Core Update 140/141 (in > February 2020) the first time. Not kept track on this, but it does not seem to cause > any harm - at least none I am aware of. But it certainly is not a good thing to see, > either... > > Please refer to https://lists.ipfire.org/pipermail/development/2020-February/007046.html > for further details. > > Thanks, and best regards, > Peter Müller > >> >> >>> On 31 Aug 2021, at 16:25, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote: >>> >>> Hi, >>> >>> On 31.08.2021 11:56, Michael Tremer wrote: >>>> Hey, >>>> >>>> This is an oops in the code that injects packets back into the kernel after they have been processed by suricata. >>> >>> Wow. How did you find this out!? >> >> There are two key functions in the trace: >> >>>>> 1 Time(s): nf_queue_entry_release_refs+0x82/0xa0 >>>>> 1 Time(s): nf_reinject+0x7a/0x1e0 >> >> They are from Netfilter and the NFQUEUE module. We only use that for the IPS. >> >>>> Was this a one-off or does this happen on a regular basis? >>> >>> Until now, it only happened once. >>> >>> It rebooted the machine - just to be sure - and its running "without >>> seen problems" since then. Absolutely normal. >> >> Would be interesting to see how it behaves if it doesn’t get a reboot. >> >> This is definitely a bug and needs to be fixed in the Linux kernel. >> >> -Michael >> >>> >>> Best, >>> Matthias >>> >>>> -Michael >>>> >>>>> On 27 Aug 2021, at 17:16, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote: >>>>> >>>>> Hi, >>>>> >>>>> today I took the usual look in LOG SUMMARY and was surprised finding this: >>>>> >>>>> ***SNIP*** >>>>> Kernel >>>>> >>>>> WARNING: Kernel Errors Present >>>>> WARNING: CPU: 0 PID: 2984 at lib/refcount.c:28 r ...: 1 Time(s) >>>>> >>>>> 1 Time(s): ? nfnetlink_net_exit_batch+0x60/0x60 >>>>> 1 Time(s): [last unloaded: hwmon_vid] >>>>> 1 Time(s): ____sys_sendmsg+0x258/0x2a0 >>>>> 1 Time(s): ___sys_sendmsg+0xa3/0xf0 >>>>> 1 Time(s): __sys_sendmsg+0x81/0xd0 >>>>> 1 Time(s): do_syscall_64+0x33/0x40 >>>>> 1 Time(s): entry_SYSCALL_64_after_hwframe+0x44/0xa9 >>>>> 1 Time(s): netlink_rcv_skb+0x5b/0x100 >>>>> 1 Time(s): netlink_sendmsg+0x23a/0x470 >>>>> 1 Time(s): netlink_unicast+0x209/0x2d0 >>>>> 1 Time(s): nf_queue_entry_release_refs+0x82/0xa0 >>>>> 1 Time(s): nf_reinject+0x7a/0x1e0 >>>>> 1 Time(s): nfnetlink_rcv_msg+0x16d/0x2c0 >>>>> 1 Time(s): nfqnl_recv_verdict+0x302/0x4f0 [nfnetlink_queue] >>>>> 1 Time(s): sock_sendmsg+0x5e/0x60 >>>>> 1 Time(s): ------------[ cut here ]------------ >>>>> 1 Time(s): ---[ end trace 49e1e291edb98731 ]--- >>>>> 1 Time(s): CPU: 0 PID: 2984 Comm: W-NFQ#1 Tainted: G O >>>>> 5.10.55-ipfire #1 >>>>> 1 Time(s): CR2: 000079fa1356a6f8 CR3: 0000000101532004 CR4: >>>>> 00000000000706f0 >>>>> 1 Time(s): CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>>> 1 Time(s): Call Trace: >>>>> 1 Time(s): Code: 05 ba 60 1f 01 01 e8 af a3 52 00 0f 0b c3 80 3d a8 60 >>>>> 1f 01 00 75 95 48 c7 c7 b8 fc 11 b4 c6 05 98 60 1f 01 01 e8 90 a3 52 00 >>>>> <0f> 0b c3 80 3d 87 60 1f 01 00 0f 85 72 ff ff ff 48 c7 c7 10 fd 11 >>>>> 1 Time(s): Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 fa ee ff >>>>> ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2e 00 00 00 0f 05 >>>>> <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 4e ef ff ff 48 >>>>> 1 Time(s): FS: 00007144d27d9640(0000) GS:ffff9db887200000(0000) >>>>> knlGS:0000000000000000 >>>>> 1 Time(s): Hardware name: To be filled by O.E.M. To be filled by >>>>> O.E.M./CRESCENTBAY, BIOS 5.011 04/13/2016 >>>>> 1 Time(s): Modules linked in: xt_REDIRECT nfnetlink_queue xt_NFQUEUE >>>>> xt_MASQUERADE ccm cpufreq_conservative cpufreq_ondemand xt_geoip(O) >>>>> xt_ipp2p(O) compat_xtables(O) xt_hashlimit xt_mac xt_multiport xt_mark >>>>> xt_policy xt_TCPMSS xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 >>>>> xt_LOG xt_limit nf_log_ipv4 nf_log_common iptable_raw iptable_mangle >>>>> iptable_filter vfat fat sch_fq_codel rt2800usb rt2x00usb >>>>> x86_pkg_temp_thermal rt2800lib intel_powerclamp coretemp rt2x00lib >>>>> kvm_intel mac80211 kvm at24 regmap_i2c iTCO_wdt iTCO_vendor_support >>>>> irqbypass crct10dif_pclmul cfg80211 crc32_pclmul ghash_clmulni_intel >>>>> pcspkr i2c_i801 rfkill r8169 lpc_ich i2c_smbus mfd_core realtek libarc4 >>>>> ir_rc6_decoder rc_rc6_mce i2c_algo_bit snd_hda_codec_realtek nuvoton_cir >>>>> fb_sys_fops syscopyarea sysfillrect sysimgblt snd_hda_codec_generic >>>>> ledtrig_audio i2c_core rc_core snd_hda_intel snd_intel_dspcfg >>>>> snd_hda_codec snd_hda_core snd_hwdep acpi_pad snd_pcm snd_timer snd >>>>> soundcore lp parport_pc parport video >>>>> 1 Time(s): R10: 0000000000000000 R11: 0000000000000293 R12: >>>>> 0000000000000000 >>>>> 1 Time(s): R10: ffffb330821af780 R11: ffffffffb4533dc8 R12: >>>>> ffff9db791371980 >>>>> 1 Time(s): R13: 0000000000000000 R14: 0000000000000001 R15: >>>>> 0000000000000070 >>>>> 1 Time(s): R13: ffff9db7444d9200 R14: 0000000000000005 R15: >>>>> ffff9db7f15c6b00 >>>>> 1 Time(s): RAX: 0000000000000000 RBX: ffff9db791371980 RCX: >>>>> 0000000000000027 >>>>> 1 Time(s): RAX: ffffffffffffffda RBX: 00007144c4268dd0 RCX: >>>>> 00007144d507062d >>>>> 1 Time(s): RBP: 00007144d27d6fe0 R08: 0000000000000000 R09: >>>>> 00007144d4754de0 >>>>> 1 Time(s): RBP: ffffb330821afa50 R08: 0000000000000000 R09: >>>>> ffffb330821af788 >>>>> 1 Time(s): RDX: 0000000000000000 RSI: 00007144d27d6f80 RDI: >>>>> 0000000000000006 >>>>> 1 Time(s): RDX: ffff9db887218968 RSI: 0000000000000001 RDI: >>>>> ffff9db887218960 >>>>> 1 Time(s): RIP: 0010:refcount_warn_saturate+0xa6/0xf0 >>>>> 1 Time(s): RIP: 0033:0x7144d507062d >>>>> 1 Time(s): RSP: 0018:ffffb330821af950 EFLAGS: 00010282 >>>>> 1 Time(s): RSP: 002b:00007144d27d6f40 EFLAGS: 00000293 ORIG_RAX: >>>>> 000000000000002e >>>>> 1 Time(s): refcount_t: underflow; use-after-free. >>>>> ***SNAP*** >>>>> >>>>> I don't know exactly how do deal with this - does anyone has an idea >>>>> what this means? >>>>> >>>>> Besides the machine is up and running with Core 159 / 64bit since 6 days >>>>> now and if I hadn't looked at the logs I would not have noticed it. >>>>> >>>>> I didn't reboot or changed anything yet- should I do? >>>>> >>>>> Best, >>>>> Matthias >>>> >>> >> > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Core 159 - "Kernel errors present" in LOG SUMMARY 2021-09-20 15:16 ` Matthias Fischer @ 2021-09-21 9:36 ` Michael Tremer 2021-11-28 14:21 ` Matthias Fischer 0 siblings, 1 reply; 8+ messages in thread From: Michael Tremer @ 2021-09-21 9:36 UTC (permalink / raw) To: development [-- Attachment #1: Type: text/plain, Size: 11151 bytes --] Hello, Could we report this to the right place with the Linux kernel community? This seems to be a long-standing problem which wouldn’t be very difficult to solve. -Michael > On 20 Sep 2021, at 16:16, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote: > > Hi, > > today it happened again - at 1:18am: > > ***SNIP*** > ipfire kernel: refcount_t: underflow; use-after-free. > ipfire kernel: WARNING: CPU: 1 PID: 30228 at lib/refcount.c:28 > refcount_warn_saturate+0xa6/0xf0 > ipfire kernel: Modules linked in: xt_REDIRECT nfnetlink_queue xt_NFQUEUE > xt_MASQUERADE ccm cpufreq_conservative cpufreq_ondemand xt_geoip(O) > xt_ipp2p(O) compat_xtables(O) xt_hashlimit xt_mac xt_multiport xt_mark > xt_policy xt_TCPMSS xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 > xt_LOG xt_limit nf_log_ipv4 nf_log_common iptable_raw iptable_mangle > iptable_filter vfat fat rt2800usb rt2x00usb rt2800lib rt2x00lib > sch_fq_codel x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel > mac80211 at24 kvm regmap_i2c iTCO_wdt iTCO_vendor_support irqbypass > crct10dif_pclmul crc32_pclmul cfg80211 ghash_clmulni_intel i2c_i801 > pcspkr i2c_smbus lpc_ich rfkill mfd_core r8169 realtek libarc4 > ir_rc6_decoder i2c_algo_bit fb_sys_fops snd_hda_codec_realtek > syscopyarea rc_rc6_mce sysfillrect snd_hda_codec_generic sysimgblt > nuvoton_cir i2c_core ledtrig_audio rc_core snd_hda_intel > snd_intel_dspcfg snd_hda_codec snd_hda_core snd_hwdep acpi_pad snd_pcm > snd_timer snd soundcore lp parport_pc parport video > ipfire kernel: CPU: 1 PID: 30228 Comm: W-NFQ#0 Tainted: G O > 5.10.55-ipfire #1 > ipfire kernel: Hardware name: To be filled by O.E.M. To be filled by > O.E.M./CRESCENTBAY, BIOS 5.011 04/13/2016 > ipfire kernel: RIP: 0010:refcount_warn_saturate+0xa6/0xf0 > ipfire kernel: Code: 05 ba 60 1f 01 01 e8 af a3 52 00 0f 0b c3 80 3d a8 > 60 1f 01 00 75 95 48 c7 c7 b8 fc 11 ab c6 05 98 60 1f 01 01 e8 90 a3 52 > 00 <0f> 0b c3 80 3d 87 60 1f 01 00 0f 85 72 ff ff ff 48 c7 c7 10 fd 11 > ipfire kernel: RSP: 0018:ffffbaba82823950 EFLAGS: 00010282 > ipfire kernel: RAX: 0000000000000000 RBX: ffff998e77a25500 RCX: > 0000000000000027 > ipfire kernel: RDX: ffff998f47318968 RSI: 0000000000000001 RDI: > ffff998f47318960 > ipfire kernel: RBP: ffffbaba82823a50 R08: 0000000000000000 R09: > ffffbaba82823788 > ipfire kernel: R10: ffffbaba82823780 R11: ffffffffab533dc8 R12: > ffff998e77a25500 > ipfire kernel: R13: ffff998e04193a00 R14: 0000000000000005 R15: > ffff998e03a6ea00 > ipfire kernel: FS: 0000758d85125640(0000) GS:ffff998f47300000(0000) > knlGS:0000000000000000 > ipfire kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > ipfire kernel: CR2: 00007722aefed0a0 CR3: 000000010351e003 CR4: > 00000000000706e0 > ipfire kernel: Call Trace: > ipfire kernel: nf_queue_entry_release_refs+0x82/0xa0 > ipfire kernel: nf_reinject+0x7a/0x1e0 > ipfire kernel: nfqnl_recv_verdict+0x302/0x4f0 [nfnetlink_queue] > ipfire kernel: nfnetlink_rcv_msg+0x16d/0x2c0 > ipfire kernel: ? nfnetlink_net_exit_batch+0x60/0x60 > ipfire kernel: netlink_rcv_skb+0x5b/0x100 > ipfire kernel: netlink_unicast+0x209/0x2d0 > ipfire kernel: netlink_sendmsg+0x23a/0x470 > ipfire kernel: sock_sendmsg+0x5e/0x60 > ipfire kernel: ____sys_sendmsg+0x258/0x2a0 > ipfire kernel: ___sys_sendmsg+0xa3/0xf0 > ipfire kernel: __sys_sendmsg+0x81/0xd0 > ipfire kernel: do_syscall_64+0x33/0x40 > ipfire kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 > ipfire kernel: RIP: 0033:0x758d871bb62d > ipfire kernel: Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 fa ee > ff ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2e 00 00 00 0f > 05 <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 4e ef ff ff 48 > ipfire kernel: RSP: 002b:0000758d85122f40 EFLAGS: 00000293 ORIG_RAX: > 000000000000002e > ipfire kernel: RAX: ffffffffffffffda RBX: 0000758d80268dd0 RCX: > 0000758d871bb62d > ipfire kernel: RDX: 0000000000000000 RSI: 0000758d85122f80 RDI: > 0000000000000005 > ipfire kernel: RBP: 0000758d85122fe0 R08: 0000000000000000 R09: > 0000758d8689fde0 > ipfire kernel: R10: 0000000000000000 R11: 0000000000000293 R12: > 0000000000000000 > ipfire kernel: R13: 0000000000000000 R14: 0000000000000001 R15: > 0000000000000070 > ipfire kernel: ---[ end trace 4c8c047c62e118e2 ]--- > ***SNAP*** > > Machine is running without (seen) problems - I didn'T reboot yet. > > Best, > Matthias > > On 09.09.2021 22:36, Peter Müller wrote: >> Hello *, >> >> just for the records: I noticed this behaviour while testing Core Update 140/141 (in >> February 2020) the first time. Not kept track on this, but it does not seem to cause >> any harm - at least none I am aware of. But it certainly is not a good thing to see, >> either... >> >> Please refer to https://lists.ipfire.org/pipermail/development/2020-February/007046.html >> for further details. >> >> Thanks, and best regards, >> Peter Müller >> >>> >>> >>>> On 31 Aug 2021, at 16:25, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote: >>>> >>>> Hi, >>>> >>>> On 31.08.2021 11:56, Michael Tremer wrote: >>>>> Hey, >>>>> >>>>> This is an oops in the code that injects packets back into the kernel after they have been processed by suricata. >>>> >>>> Wow. How did you find this out!? >>> >>> There are two key functions in the trace: >>> >>>>>> 1 Time(s): nf_queue_entry_release_refs+0x82/0xa0 >>>>>> 1 Time(s): nf_reinject+0x7a/0x1e0 >>> >>> They are from Netfilter and the NFQUEUE module. We only use that for the IPS. >>> >>>>> Was this a one-off or does this happen on a regular basis? >>>> >>>> Until now, it only happened once. >>>> >>>> It rebooted the machine - just to be sure - and its running "without >>>> seen problems" since then. Absolutely normal. >>> >>> Would be interesting to see how it behaves if it doesn’t get a reboot. >>> >>> This is definitely a bug and needs to be fixed in the Linux kernel. >>> >>> -Michael >>> >>>> >>>> Best, >>>> Matthias >>>> >>>>> -Michael >>>>> >>>>>> On 27 Aug 2021, at 17:16, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> today I took the usual look in LOG SUMMARY and was surprised finding this: >>>>>> >>>>>> ***SNIP*** >>>>>> Kernel >>>>>> >>>>>> WARNING: Kernel Errors Present >>>>>> WARNING: CPU: 0 PID: 2984 at lib/refcount.c:28 r ...: 1 Time(s) >>>>>> >>>>>> 1 Time(s): ? nfnetlink_net_exit_batch+0x60/0x60 >>>>>> 1 Time(s): [last unloaded: hwmon_vid] >>>>>> 1 Time(s): ____sys_sendmsg+0x258/0x2a0 >>>>>> 1 Time(s): ___sys_sendmsg+0xa3/0xf0 >>>>>> 1 Time(s): __sys_sendmsg+0x81/0xd0 >>>>>> 1 Time(s): do_syscall_64+0x33/0x40 >>>>>> 1 Time(s): entry_SYSCALL_64_after_hwframe+0x44/0xa9 >>>>>> 1 Time(s): netlink_rcv_skb+0x5b/0x100 >>>>>> 1 Time(s): netlink_sendmsg+0x23a/0x470 >>>>>> 1 Time(s): netlink_unicast+0x209/0x2d0 >>>>>> 1 Time(s): nf_queue_entry_release_refs+0x82/0xa0 >>>>>> 1 Time(s): nf_reinject+0x7a/0x1e0 >>>>>> 1 Time(s): nfnetlink_rcv_msg+0x16d/0x2c0 >>>>>> 1 Time(s): nfqnl_recv_verdict+0x302/0x4f0 [nfnetlink_queue] >>>>>> 1 Time(s): sock_sendmsg+0x5e/0x60 >>>>>> 1 Time(s): ------------[ cut here ]------------ >>>>>> 1 Time(s): ---[ end trace 49e1e291edb98731 ]--- >>>>>> 1 Time(s): CPU: 0 PID: 2984 Comm: W-NFQ#1 Tainted: G O >>>>>> 5.10.55-ipfire #1 >>>>>> 1 Time(s): CR2: 000079fa1356a6f8 CR3: 0000000101532004 CR4: >>>>>> 00000000000706f0 >>>>>> 1 Time(s): CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>>>> 1 Time(s): Call Trace: >>>>>> 1 Time(s): Code: 05 ba 60 1f 01 01 e8 af a3 52 00 0f 0b c3 80 3d a8 60 >>>>>> 1f 01 00 75 95 48 c7 c7 b8 fc 11 b4 c6 05 98 60 1f 01 01 e8 90 a3 52 00 >>>>>> <0f> 0b c3 80 3d 87 60 1f 01 00 0f 85 72 ff ff ff 48 c7 c7 10 fd 11 >>>>>> 1 Time(s): Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 fa ee ff >>>>>> ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2e 00 00 00 0f 05 >>>>>> <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 4e ef ff ff 48 >>>>>> 1 Time(s): FS: 00007144d27d9640(0000) GS:ffff9db887200000(0000) >>>>>> knlGS:0000000000000000 >>>>>> 1 Time(s): Hardware name: To be filled by O.E.M. To be filled by >>>>>> O.E.M./CRESCENTBAY, BIOS 5.011 04/13/2016 >>>>>> 1 Time(s): Modules linked in: xt_REDIRECT nfnetlink_queue xt_NFQUEUE >>>>>> xt_MASQUERADE ccm cpufreq_conservative cpufreq_ondemand xt_geoip(O) >>>>>> xt_ipp2p(O) compat_xtables(O) xt_hashlimit xt_mac xt_multiport xt_mark >>>>>> xt_policy xt_TCPMSS xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 >>>>>> xt_LOG xt_limit nf_log_ipv4 nf_log_common iptable_raw iptable_mangle >>>>>> iptable_filter vfat fat sch_fq_codel rt2800usb rt2x00usb >>>>>> x86_pkg_temp_thermal rt2800lib intel_powerclamp coretemp rt2x00lib >>>>>> kvm_intel mac80211 kvm at24 regmap_i2c iTCO_wdt iTCO_vendor_support >>>>>> irqbypass crct10dif_pclmul cfg80211 crc32_pclmul ghash_clmulni_intel >>>>>> pcspkr i2c_i801 rfkill r8169 lpc_ich i2c_smbus mfd_core realtek libarc4 >>>>>> ir_rc6_decoder rc_rc6_mce i2c_algo_bit snd_hda_codec_realtek nuvoton_cir >>>>>> fb_sys_fops syscopyarea sysfillrect sysimgblt snd_hda_codec_generic >>>>>> ledtrig_audio i2c_core rc_core snd_hda_intel snd_intel_dspcfg >>>>>> snd_hda_codec snd_hda_core snd_hwdep acpi_pad snd_pcm snd_timer snd >>>>>> soundcore lp parport_pc parport video >>>>>> 1 Time(s): R10: 0000000000000000 R11: 0000000000000293 R12: >>>>>> 0000000000000000 >>>>>> 1 Time(s): R10: ffffb330821af780 R11: ffffffffb4533dc8 R12: >>>>>> ffff9db791371980 >>>>>> 1 Time(s): R13: 0000000000000000 R14: 0000000000000001 R15: >>>>>> 0000000000000070 >>>>>> 1 Time(s): R13: ffff9db7444d9200 R14: 0000000000000005 R15: >>>>>> ffff9db7f15c6b00 >>>>>> 1 Time(s): RAX: 0000000000000000 RBX: ffff9db791371980 RCX: >>>>>> 0000000000000027 >>>>>> 1 Time(s): RAX: ffffffffffffffda RBX: 00007144c4268dd0 RCX: >>>>>> 00007144d507062d >>>>>> 1 Time(s): RBP: 00007144d27d6fe0 R08: 0000000000000000 R09: >>>>>> 00007144d4754de0 >>>>>> 1 Time(s): RBP: ffffb330821afa50 R08: 0000000000000000 R09: >>>>>> ffffb330821af788 >>>>>> 1 Time(s): RDX: 0000000000000000 RSI: 00007144d27d6f80 RDI: >>>>>> 0000000000000006 >>>>>> 1 Time(s): RDX: ffff9db887218968 RSI: 0000000000000001 RDI: >>>>>> ffff9db887218960 >>>>>> 1 Time(s): RIP: 0010:refcount_warn_saturate+0xa6/0xf0 >>>>>> 1 Time(s): RIP: 0033:0x7144d507062d >>>>>> 1 Time(s): RSP: 0018:ffffb330821af950 EFLAGS: 00010282 >>>>>> 1 Time(s): RSP: 002b:00007144d27d6f40 EFLAGS: 00000293 ORIG_RAX: >>>>>> 000000000000002e >>>>>> 1 Time(s): refcount_t: underflow; use-after-free. >>>>>> ***SNAP*** >>>>>> >>>>>> I don't know exactly how do deal with this - does anyone has an idea >>>>>> what this means? >>>>>> >>>>>> Besides the machine is up and running with Core 159 / 64bit since 6 days >>>>>> now and if I hadn't looked at the logs I would not have noticed it. >>>>>> >>>>>> I didn't reboot or changed anything yet- should I do? >>>>>> >>>>>> Best, >>>>>> Matthias >>>>> >>>> >>> >> > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Core 159 - "Kernel errors present" in LOG SUMMARY 2021-09-21 9:36 ` Michael Tremer @ 2021-11-28 14:21 ` Matthias Fischer 2021-11-28 17:03 ` Michael Tremer 0 siblings, 1 reply; 8+ messages in thread From: Matthias Fischer @ 2021-11-28 14:21 UTC (permalink / raw) To: development [-- Attachment #1: Type: text/plain, Size: 11667 bytes --] Hi, On 21.09.2021 11:36, Michael Tremer wrote: > Hello, > > Could we report this to the right place with the Linux kernel community? Where would be this "right place"? https://bugzilla.kernel.org/ or https://lkml.org/? It happened again today with Core 161 (running 'suricata 5.0.8'). => see Attachment Best, Matthias > This seems to be a long-standing problem which wouldn’t be very difficult to solve. > > -Michael > >> On 20 Sep 2021, at 16:16, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote: >> >> Hi, >> >> today it happened again - at 1:18am: >> >> ***SNIP*** >> ipfire kernel: refcount_t: underflow; use-after-free. >> ipfire kernel: WARNING: CPU: 1 PID: 30228 at lib/refcount.c:28 >> refcount_warn_saturate+0xa6/0xf0 >> ipfire kernel: Modules linked in: xt_REDIRECT nfnetlink_queue xt_NFQUEUE >> xt_MASQUERADE ccm cpufreq_conservative cpufreq_ondemand xt_geoip(O) >> xt_ipp2p(O) compat_xtables(O) xt_hashlimit xt_mac xt_multiport xt_mark >> xt_policy xt_TCPMSS xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 >> xt_LOG xt_limit nf_log_ipv4 nf_log_common iptable_raw iptable_mangle >> iptable_filter vfat fat rt2800usb rt2x00usb rt2800lib rt2x00lib >> sch_fq_codel x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel >> mac80211 at24 kvm regmap_i2c iTCO_wdt iTCO_vendor_support irqbypass >> crct10dif_pclmul crc32_pclmul cfg80211 ghash_clmulni_intel i2c_i801 >> pcspkr i2c_smbus lpc_ich rfkill mfd_core r8169 realtek libarc4 >> ir_rc6_decoder i2c_algo_bit fb_sys_fops snd_hda_codec_realtek >> syscopyarea rc_rc6_mce sysfillrect snd_hda_codec_generic sysimgblt >> nuvoton_cir i2c_core ledtrig_audio rc_core snd_hda_intel >> snd_intel_dspcfg snd_hda_codec snd_hda_core snd_hwdep acpi_pad snd_pcm >> snd_timer snd soundcore lp parport_pc parport video >> ipfire kernel: CPU: 1 PID: 30228 Comm: W-NFQ#0 Tainted: G O >> 5.10.55-ipfire #1 >> ipfire kernel: Hardware name: To be filled by O.E.M. To be filled by >> O.E.M./CRESCENTBAY, BIOS 5.011 04/13/2016 >> ipfire kernel: RIP: 0010:refcount_warn_saturate+0xa6/0xf0 >> ipfire kernel: Code: 05 ba 60 1f 01 01 e8 af a3 52 00 0f 0b c3 80 3d a8 >> 60 1f 01 00 75 95 48 c7 c7 b8 fc 11 ab c6 05 98 60 1f 01 01 e8 90 a3 52 >> 00 <0f> 0b c3 80 3d 87 60 1f 01 00 0f 85 72 ff ff ff 48 c7 c7 10 fd 11 >> ipfire kernel: RSP: 0018:ffffbaba82823950 EFLAGS: 00010282 >> ipfire kernel: RAX: 0000000000000000 RBX: ffff998e77a25500 RCX: >> 0000000000000027 >> ipfire kernel: RDX: ffff998f47318968 RSI: 0000000000000001 RDI: >> ffff998f47318960 >> ipfire kernel: RBP: ffffbaba82823a50 R08: 0000000000000000 R09: >> ffffbaba82823788 >> ipfire kernel: R10: ffffbaba82823780 R11: ffffffffab533dc8 R12: >> ffff998e77a25500 >> ipfire kernel: R13: ffff998e04193a00 R14: 0000000000000005 R15: >> ffff998e03a6ea00 >> ipfire kernel: FS: 0000758d85125640(0000) GS:ffff998f47300000(0000) >> knlGS:0000000000000000 >> ipfire kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> ipfire kernel: CR2: 00007722aefed0a0 CR3: 000000010351e003 CR4: >> 00000000000706e0 >> ipfire kernel: Call Trace: >> ipfire kernel: nf_queue_entry_release_refs+0x82/0xa0 >> ipfire kernel: nf_reinject+0x7a/0x1e0 >> ipfire kernel: nfqnl_recv_verdict+0x302/0x4f0 [nfnetlink_queue] >> ipfire kernel: nfnetlink_rcv_msg+0x16d/0x2c0 >> ipfire kernel: ? nfnetlink_net_exit_batch+0x60/0x60 >> ipfire kernel: netlink_rcv_skb+0x5b/0x100 >> ipfire kernel: netlink_unicast+0x209/0x2d0 >> ipfire kernel: netlink_sendmsg+0x23a/0x470 >> ipfire kernel: sock_sendmsg+0x5e/0x60 >> ipfire kernel: ____sys_sendmsg+0x258/0x2a0 >> ipfire kernel: ___sys_sendmsg+0xa3/0xf0 >> ipfire kernel: __sys_sendmsg+0x81/0xd0 >> ipfire kernel: do_syscall_64+0x33/0x40 >> ipfire kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 >> ipfire kernel: RIP: 0033:0x758d871bb62d >> ipfire kernel: Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 fa ee >> ff ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2e 00 00 00 0f >> 05 <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 4e ef ff ff 48 >> ipfire kernel: RSP: 002b:0000758d85122f40 EFLAGS: 00000293 ORIG_RAX: >> 000000000000002e >> ipfire kernel: RAX: ffffffffffffffda RBX: 0000758d80268dd0 RCX: >> 0000758d871bb62d >> ipfire kernel: RDX: 0000000000000000 RSI: 0000758d85122f80 RDI: >> 0000000000000005 >> ipfire kernel: RBP: 0000758d85122fe0 R08: 0000000000000000 R09: >> 0000758d8689fde0 >> ipfire kernel: R10: 0000000000000000 R11: 0000000000000293 R12: >> 0000000000000000 >> ipfire kernel: R13: 0000000000000000 R14: 0000000000000001 R15: >> 0000000000000070 >> ipfire kernel: ---[ end trace 4c8c047c62e118e2 ]--- >> ***SNAP*** >> >> Machine is running without (seen) problems - I didn'T reboot yet. >> >> Best, >> Matthias >> >> On 09.09.2021 22:36, Peter Müller wrote: >>> Hello *, >>> >>> just for the records: I noticed this behaviour while testing Core Update 140/141 (in >>> February 2020) the first time. Not kept track on this, but it does not seem to cause >>> any harm - at least none I am aware of. But it certainly is not a good thing to see, >>> either... >>> >>> Please refer to https://lists.ipfire.org/pipermail/development/2020-February/007046.html >>> for further details. >>> >>> Thanks, and best regards, >>> Peter Müller >>> >>>> >>>> >>>>> On 31 Aug 2021, at 16:25, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote: >>>>> >>>>> Hi, >>>>> >>>>> On 31.08.2021 11:56, Michael Tremer wrote: >>>>>> Hey, >>>>>> >>>>>> This is an oops in the code that injects packets back into the kernel after they have been processed by suricata. >>>>> >>>>> Wow. How did you find this out!? >>>> >>>> There are two key functions in the trace: >>>> >>>>>>> 1 Time(s): nf_queue_entry_release_refs+0x82/0xa0 >>>>>>> 1 Time(s): nf_reinject+0x7a/0x1e0 >>>> >>>> They are from Netfilter and the NFQUEUE module. We only use that for the IPS. >>>> >>>>>> Was this a one-off or does this happen on a regular basis? >>>>> >>>>> Until now, it only happened once. >>>>> >>>>> It rebooted the machine - just to be sure - and its running "without >>>>> seen problems" since then. Absolutely normal. >>>> >>>> Would be interesting to see how it behaves if it doesn’t get a reboot. >>>> >>>> This is definitely a bug and needs to be fixed in the Linux kernel. >>>> >>>> -Michael >>>> >>>>> >>>>> Best, >>>>> Matthias >>>>> >>>>>> -Michael >>>>>> >>>>>>> On 27 Aug 2021, at 17:16, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> today I took the usual look in LOG SUMMARY and was surprised finding this: >>>>>>> >>>>>>> ***SNIP*** >>>>>>> Kernel >>>>>>> >>>>>>> WARNING: Kernel Errors Present >>>>>>> WARNING: CPU: 0 PID: 2984 at lib/refcount.c:28 r ...: 1 Time(s) >>>>>>> >>>>>>> 1 Time(s): ? nfnetlink_net_exit_batch+0x60/0x60 >>>>>>> 1 Time(s): [last unloaded: hwmon_vid] >>>>>>> 1 Time(s): ____sys_sendmsg+0x258/0x2a0 >>>>>>> 1 Time(s): ___sys_sendmsg+0xa3/0xf0 >>>>>>> 1 Time(s): __sys_sendmsg+0x81/0xd0 >>>>>>> 1 Time(s): do_syscall_64+0x33/0x40 >>>>>>> 1 Time(s): entry_SYSCALL_64_after_hwframe+0x44/0xa9 >>>>>>> 1 Time(s): netlink_rcv_skb+0x5b/0x100 >>>>>>> 1 Time(s): netlink_sendmsg+0x23a/0x470 >>>>>>> 1 Time(s): netlink_unicast+0x209/0x2d0 >>>>>>> 1 Time(s): nf_queue_entry_release_refs+0x82/0xa0 >>>>>>> 1 Time(s): nf_reinject+0x7a/0x1e0 >>>>>>> 1 Time(s): nfnetlink_rcv_msg+0x16d/0x2c0 >>>>>>> 1 Time(s): nfqnl_recv_verdict+0x302/0x4f0 [nfnetlink_queue] >>>>>>> 1 Time(s): sock_sendmsg+0x5e/0x60 >>>>>>> 1 Time(s): ------------[ cut here ]------------ >>>>>>> 1 Time(s): ---[ end trace 49e1e291edb98731 ]--- >>>>>>> 1 Time(s): CPU: 0 PID: 2984 Comm: W-NFQ#1 Tainted: G O >>>>>>> 5.10.55-ipfire #1 >>>>>>> 1 Time(s): CR2: 000079fa1356a6f8 CR3: 0000000101532004 CR4: >>>>>>> 00000000000706f0 >>>>>>> 1 Time(s): CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>>>>> 1 Time(s): Call Trace: >>>>>>> 1 Time(s): Code: 05 ba 60 1f 01 01 e8 af a3 52 00 0f 0b c3 80 3d a8 60 >>>>>>> 1f 01 00 75 95 48 c7 c7 b8 fc 11 b4 c6 05 98 60 1f 01 01 e8 90 a3 52 00 >>>>>>> <0f> 0b c3 80 3d 87 60 1f 01 00 0f 85 72 ff ff ff 48 c7 c7 10 fd 11 >>>>>>> 1 Time(s): Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 fa ee ff >>>>>>> ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2e 00 00 00 0f 05 >>>>>>> <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 4e ef ff ff 48 >>>>>>> 1 Time(s): FS: 00007144d27d9640(0000) GS:ffff9db887200000(0000) >>>>>>> knlGS:0000000000000000 >>>>>>> 1 Time(s): Hardware name: To be filled by O.E.M. To be filled by >>>>>>> O.E.M./CRESCENTBAY, BIOS 5.011 04/13/2016 >>>>>>> 1 Time(s): Modules linked in: xt_REDIRECT nfnetlink_queue xt_NFQUEUE >>>>>>> xt_MASQUERADE ccm cpufreq_conservative cpufreq_ondemand xt_geoip(O) >>>>>>> xt_ipp2p(O) compat_xtables(O) xt_hashlimit xt_mac xt_multiport xt_mark >>>>>>> xt_policy xt_TCPMSS xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 >>>>>>> xt_LOG xt_limit nf_log_ipv4 nf_log_common iptable_raw iptable_mangle >>>>>>> iptable_filter vfat fat sch_fq_codel rt2800usb rt2x00usb >>>>>>> x86_pkg_temp_thermal rt2800lib intel_powerclamp coretemp rt2x00lib >>>>>>> kvm_intel mac80211 kvm at24 regmap_i2c iTCO_wdt iTCO_vendor_support >>>>>>> irqbypass crct10dif_pclmul cfg80211 crc32_pclmul ghash_clmulni_intel >>>>>>> pcspkr i2c_i801 rfkill r8169 lpc_ich i2c_smbus mfd_core realtek libarc4 >>>>>>> ir_rc6_decoder rc_rc6_mce i2c_algo_bit snd_hda_codec_realtek nuvoton_cir >>>>>>> fb_sys_fops syscopyarea sysfillrect sysimgblt snd_hda_codec_generic >>>>>>> ledtrig_audio i2c_core rc_core snd_hda_intel snd_intel_dspcfg >>>>>>> snd_hda_codec snd_hda_core snd_hwdep acpi_pad snd_pcm snd_timer snd >>>>>>> soundcore lp parport_pc parport video >>>>>>> 1 Time(s): R10: 0000000000000000 R11: 0000000000000293 R12: >>>>>>> 0000000000000000 >>>>>>> 1 Time(s): R10: ffffb330821af780 R11: ffffffffb4533dc8 R12: >>>>>>> ffff9db791371980 >>>>>>> 1 Time(s): R13: 0000000000000000 R14: 0000000000000001 R15: >>>>>>> 0000000000000070 >>>>>>> 1 Time(s): R13: ffff9db7444d9200 R14: 0000000000000005 R15: >>>>>>> ffff9db7f15c6b00 >>>>>>> 1 Time(s): RAX: 0000000000000000 RBX: ffff9db791371980 RCX: >>>>>>> 0000000000000027 >>>>>>> 1 Time(s): RAX: ffffffffffffffda RBX: 00007144c4268dd0 RCX: >>>>>>> 00007144d507062d >>>>>>> 1 Time(s): RBP: 00007144d27d6fe0 R08: 0000000000000000 R09: >>>>>>> 00007144d4754de0 >>>>>>> 1 Time(s): RBP: ffffb330821afa50 R08: 0000000000000000 R09: >>>>>>> ffffb330821af788 >>>>>>> 1 Time(s): RDX: 0000000000000000 RSI: 00007144d27d6f80 RDI: >>>>>>> 0000000000000006 >>>>>>> 1 Time(s): RDX: ffff9db887218968 RSI: 0000000000000001 RDI: >>>>>>> ffff9db887218960 >>>>>>> 1 Time(s): RIP: 0010:refcount_warn_saturate+0xa6/0xf0 >>>>>>> 1 Time(s): RIP: 0033:0x7144d507062d >>>>>>> 1 Time(s): RSP: 0018:ffffb330821af950 EFLAGS: 00010282 >>>>>>> 1 Time(s): RSP: 002b:00007144d27d6f40 EFLAGS: 00000293 ORIG_RAX: >>>>>>> 000000000000002e >>>>>>> 1 Time(s): refcount_t: underflow; use-after-free. >>>>>>> ***SNAP*** >>>>>>> >>>>>>> I don't know exactly how do deal with this - does anyone has an idea >>>>>>> what this means? >>>>>>> >>>>>>> Besides the machine is up and running with Core 159 / 64bit since 6 days >>>>>>> now and if I hadn't looked at the logs I would not have noticed it. >>>>>>> >>>>>>> I didn't reboot or changed anything yet- should I do? >>>>>>> >>>>>>> Best, >>>>>>> Matthias >>>>>> >>>>> >>>> >>> >> > [-- Attachment #2: kernelerror-excerptmessages_02.txt --] [-- Type: text/plain, Size: 4917 bytes --] Nov 28 14:46:24 ipfire kernel: ------------[ cut here ]------------ Nov 28 14:46:24 ipfire kernel: refcount_t: underflow; use-after-free. Nov 28 14:46:24 ipfire kernel: WARNING: CPU: 1 PID: 26578 at lib/refcount.c:28 refcount_warn_saturate+0xa6/0xf0 Nov 28 14:46:24 ipfire kernel: Modules linked in: nct6775 hwmon_vid nfnetlink_queue xt_NFQUEUE xt_MASQUERADE ccm cpufreq_conservative cpufreq_ondemand xt_geoip(O) xt_ipp2p(O) compat_xtables(O) xt_REDIRECT xt_hashlimit xt_mac xt_multiport xt_policy xt_TCPMSS xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 xt_LOG xt_limit xt_mark xt_connmark nf_log_ipv4 nf_log_common iptable_raw iptable_mangle iptable_filter vfat fat rt2800usb rt2x00usb rt2800lib sch_fq_codel rt2x00lib mac80211 x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul at24 cfg80211 regmap_i2c iTCO_wdt iTCO_vendor_support snd_hda_codec_realtek snd_hda_codec_generic i2c_algo_bit fb_sys_fops ledtrig_audio syscopyarea sysfillrect sysimgblt snd_hda_intel snd_intel_dspcfg snd_hda_codec i2c_i801 ghash_clmulni_intel r8169 i2c_smbus snd_hda_core i2c_core pcspkr rfkill lpc_ich libarc4 realtek ir_rc6_decoder mfd_core snd_hwdep rc_rc6_mce snd_pcm nuvoton_cir rc_core snd_timer snd acpi_pad soundcore lp Nov 28 14:46:24 ipfire kernel: parport_pc parport video [last unloaded: i2c_dev] Nov 28 14:46:24 ipfire kernel: CPU: 1 PID: 26578 Comm: W-NFQ#0 Tainted: G O 5.10.76-ipfire #1 Nov 28 14:46:24 ipfire kernel: Hardware name: To be filled by O.E.M. To be filled by O.E.M./CRESCENTBAY, BIOS 5.011 04/13/2016 Nov 28 14:46:24 ipfire kernel: RIP: 0010:refcount_warn_saturate+0xa6/0xf0 Nov 28 14:46:24 ipfire kernel: Code: 05 ad 6f 1e 01 01 e8 d6 98 52 00 0f 0b c3 80 3d 9b 6f 1e 01 00 75 95 48 c7 c7 b8 1c d2 96 c6 05 8b 6f 1e 01 01 e8 b7 98 52 00 <0f> 0b c3 80 3d 7a 6f 1e 01 00 0f 85 72 ff ff ff 48 c7 c7 10 1d d2 Nov 28 14:46:24 ipfire kernel: RSP: 0018:ffff915c83bcf968 EFLAGS: 00010286 Nov 28 14:46:24 ipfire kernel: RAX: 0000000000000000 RBX: ffff8f613ad71c00 RCX: 0000000000000027 Nov 28 14:46:24 ipfire kernel: RDX: ffff8f61873189c8 RSI: 0000000000000001 RDI: ffff8f61873189c0 Nov 28 14:46:24 ipfire kernel: RBP: ffff915c83bcfa68 R08: 0000000000000000 R09: ffff915c83bcf7a0 Nov 28 14:46:24 ipfire kernel: R10: ffff915c83bcf798 R11: ffffffff97133f48 R12: ffff8f613ad71c00 Nov 28 14:46:24 ipfire kernel: R13: ffff8f60d0a76000 R14: 0000000000000005 R15: ffff8f604303b500 Nov 28 14:46:24 ipfire kernel: FS: 00007aed0c476640(0000) GS:ffff8f6187300000(0000) knlGS:0000000000000000 Nov 28 14:46:24 ipfire kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 28 14:46:24 ipfire kernel: CR2: 0000760e4b48c000 CR3: 000000010411c003 CR4: 00000000000706e0 Nov 28 14:46:24 ipfire kernel: Call Trace: Nov 28 14:46:24 ipfire kernel: nf_queue_entry_release_refs+0x82/0xa0 Nov 28 14:46:24 ipfire kernel: nf_reinject+0x7a/0x1e0 Nov 28 14:46:24 ipfire kernel: nfqnl_recv_verdict+0x302/0x4f0 [nfnetlink_queue] Nov 28 14:46:24 ipfire kernel: nfnetlink_rcv_msg+0x16d/0x2c0 Nov 28 14:46:24 ipfire kernel: ? nfnetlink_net_exit_batch+0x60/0x60 Nov 28 14:46:24 ipfire kernel: netlink_rcv_skb+0x5b/0x100 Nov 28 14:46:24 ipfire kernel: netlink_unicast+0x209/0x2d0 Nov 28 14:46:24 ipfire kernel: netlink_sendmsg+0x22d/0x460 Nov 28 14:46:24 ipfire kernel: ? netlink_unicast+0x2d0/0x2d0 Nov 28 14:46:24 ipfire kernel: ____sys_sendmsg+0x298/0x2d0 Nov 28 14:46:24 ipfire kernel: ___sys_sendmsg+0xa3/0xf0 Nov 28 14:46:24 ipfire kernel: __sys_sendmsg+0x81/0xd0 Nov 28 14:46:24 ipfire kernel: do_syscall_64+0x33/0x40 Nov 28 14:46:24 ipfire kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Nov 28 14:46:24 ipfire kernel: RIP: 0033:0x7aed0e51162d Nov 28 14:46:24 ipfire kernel: Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 fa ee ff ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 4e ef ff ff 48 Nov 28 14:46:24 ipfire kernel: RSP: 002b:00007aed0c473f40 EFLAGS: 00000293 ORIG_RAX: 000000000000002e Nov 28 14:46:24 ipfire kernel: RAX: ffffffffffffffda RBX: 00007aed04268dd0 RCX: 00007aed0e51162d Nov 28 14:46:24 ipfire kernel: RDX: 0000000000000000 RSI: 00007aed0c473f80 RDI: 0000000000000005 Nov 28 14:46:24 ipfire kernel: RBP: 00007aed0c473fe0 R08: 0000000000000000 R09: 00007aed0dbf0de0 Nov 28 14:46:24 ipfire kernel: R10: 00007aecfc444160 R11: 0000000000000293 R12: 0000000000000000 Nov 28 14:46:24 ipfire kernel: R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000080 Nov 28 14:46:24 ipfire kernel: ---[ end trace 8bef469b91aea880 ]--- Nov 28 14:46:30 ipfire monit[3814]: 'trace' content match: Nov 28 14:46:24 ipfire kernel: ---[ end trace 8bef469b91aea880 ]--- Nov 28 14:47:31 ipfire monit[3814]: 'trace' content match: Nov 28 14:46:30 ipfire monit[3814]: 'trace' content match: Nov 28 14:46:24 ipfire kernel: ---[ end trace 8bef469b91aea880 ]--- ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Core 159 - "Kernel errors present" in LOG SUMMARY 2021-11-28 14:21 ` Matthias Fischer @ 2021-11-28 17:03 ` Michael Tremer 2021-12-27 23:49 ` Matthias Fischer 0 siblings, 1 reply; 8+ messages in thread From: Michael Tremer @ 2021-11-28 17:03 UTC (permalink / raw) To: development [-- Attachment #1: Type: text/plain, Size: 12229 bytes --] Hello Matthias, Since we now have kernel 5.15.x in next, I would recommend installing that and check if the problem persists. Best, -Michael > On 28 Nov 2021, at 14:21, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote: > > Hi, > > On 21.09.2021 11:36, Michael Tremer wrote: >> Hello, >> >> Could we report this to the right place with the Linux kernel community? > > Where would be this "right place"? > > https://bugzilla.kernel.org/ or https://lkml.org/? > > It happened again today with Core 161 (running 'suricata 5.0.8'). > > => see Attachment > > Best, > Matthias > >> This seems to be a long-standing problem which wouldn’t be very difficult to solve. >> >> -Michael >> >>> On 20 Sep 2021, at 16:16, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote: >>> >>> Hi, >>> >>> today it happened again - at 1:18am: >>> >>> ***SNIP*** >>> ipfire kernel: refcount_t: underflow; use-after-free. >>> ipfire kernel: WARNING: CPU: 1 PID: 30228 at lib/refcount.c:28 >>> refcount_warn_saturate+0xa6/0xf0 >>> ipfire kernel: Modules linked in: xt_REDIRECT nfnetlink_queue xt_NFQUEUE >>> xt_MASQUERADE ccm cpufreq_conservative cpufreq_ondemand xt_geoip(O) >>> xt_ipp2p(O) compat_xtables(O) xt_hashlimit xt_mac xt_multiport xt_mark >>> xt_policy xt_TCPMSS xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 >>> xt_LOG xt_limit nf_log_ipv4 nf_log_common iptable_raw iptable_mangle >>> iptable_filter vfat fat rt2800usb rt2x00usb rt2800lib rt2x00lib >>> sch_fq_codel x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel >>> mac80211 at24 kvm regmap_i2c iTCO_wdt iTCO_vendor_support irqbypass >>> crct10dif_pclmul crc32_pclmul cfg80211 ghash_clmulni_intel i2c_i801 >>> pcspkr i2c_smbus lpc_ich rfkill mfd_core r8169 realtek libarc4 >>> ir_rc6_decoder i2c_algo_bit fb_sys_fops snd_hda_codec_realtek >>> syscopyarea rc_rc6_mce sysfillrect snd_hda_codec_generic sysimgblt >>> nuvoton_cir i2c_core ledtrig_audio rc_core snd_hda_intel >>> snd_intel_dspcfg snd_hda_codec snd_hda_core snd_hwdep acpi_pad snd_pcm >>> snd_timer snd soundcore lp parport_pc parport video >>> ipfire kernel: CPU: 1 PID: 30228 Comm: W-NFQ#0 Tainted: G O >>> 5.10.55-ipfire #1 >>> ipfire kernel: Hardware name: To be filled by O.E.M. To be filled by >>> O.E.M./CRESCENTBAY, BIOS 5.011 04/13/2016 >>> ipfire kernel: RIP: 0010:refcount_warn_saturate+0xa6/0xf0 >>> ipfire kernel: Code: 05 ba 60 1f 01 01 e8 af a3 52 00 0f 0b c3 80 3d a8 >>> 60 1f 01 00 75 95 48 c7 c7 b8 fc 11 ab c6 05 98 60 1f 01 01 e8 90 a3 52 >>> 00 <0f> 0b c3 80 3d 87 60 1f 01 00 0f 85 72 ff ff ff 48 c7 c7 10 fd 11 >>> ipfire kernel: RSP: 0018:ffffbaba82823950 EFLAGS: 00010282 >>> ipfire kernel: RAX: 0000000000000000 RBX: ffff998e77a25500 RCX: >>> 0000000000000027 >>> ipfire kernel: RDX: ffff998f47318968 RSI: 0000000000000001 RDI: >>> ffff998f47318960 >>> ipfire kernel: RBP: ffffbaba82823a50 R08: 0000000000000000 R09: >>> ffffbaba82823788 >>> ipfire kernel: R10: ffffbaba82823780 R11: ffffffffab533dc8 R12: >>> ffff998e77a25500 >>> ipfire kernel: R13: ffff998e04193a00 R14: 0000000000000005 R15: >>> ffff998e03a6ea00 >>> ipfire kernel: FS: 0000758d85125640(0000) GS:ffff998f47300000(0000) >>> knlGS:0000000000000000 >>> ipfire kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>> ipfire kernel: CR2: 00007722aefed0a0 CR3: 000000010351e003 CR4: >>> 00000000000706e0 >>> ipfire kernel: Call Trace: >>> ipfire kernel: nf_queue_entry_release_refs+0x82/0xa0 >>> ipfire kernel: nf_reinject+0x7a/0x1e0 >>> ipfire kernel: nfqnl_recv_verdict+0x302/0x4f0 [nfnetlink_queue] >>> ipfire kernel: nfnetlink_rcv_msg+0x16d/0x2c0 >>> ipfire kernel: ? nfnetlink_net_exit_batch+0x60/0x60 >>> ipfire kernel: netlink_rcv_skb+0x5b/0x100 >>> ipfire kernel: netlink_unicast+0x209/0x2d0 >>> ipfire kernel: netlink_sendmsg+0x23a/0x470 >>> ipfire kernel: sock_sendmsg+0x5e/0x60 >>> ipfire kernel: ____sys_sendmsg+0x258/0x2a0 >>> ipfire kernel: ___sys_sendmsg+0xa3/0xf0 >>> ipfire kernel: __sys_sendmsg+0x81/0xd0 >>> ipfire kernel: do_syscall_64+0x33/0x40 >>> ipfire kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 >>> ipfire kernel: RIP: 0033:0x758d871bb62d >>> ipfire kernel: Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 fa ee >>> ff ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2e 00 00 00 0f >>> 05 <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 4e ef ff ff 48 >>> ipfire kernel: RSP: 002b:0000758d85122f40 EFLAGS: 00000293 ORIG_RAX: >>> 000000000000002e >>> ipfire kernel: RAX: ffffffffffffffda RBX: 0000758d80268dd0 RCX: >>> 0000758d871bb62d >>> ipfire kernel: RDX: 0000000000000000 RSI: 0000758d85122f80 RDI: >>> 0000000000000005 >>> ipfire kernel: RBP: 0000758d85122fe0 R08: 0000000000000000 R09: >>> 0000758d8689fde0 >>> ipfire kernel: R10: 0000000000000000 R11: 0000000000000293 R12: >>> 0000000000000000 >>> ipfire kernel: R13: 0000000000000000 R14: 0000000000000001 R15: >>> 0000000000000070 >>> ipfire kernel: ---[ end trace 4c8c047c62e118e2 ]--- >>> ***SNAP*** >>> >>> Machine is running without (seen) problems - I didn'T reboot yet. >>> >>> Best, >>> Matthias >>> >>> On 09.09.2021 22:36, Peter Müller wrote: >>>> Hello *, >>>> >>>> just for the records: I noticed this behaviour while testing Core Update 140/141 (in >>>> February 2020) the first time. Not kept track on this, but it does not seem to cause >>>> any harm - at least none I am aware of. But it certainly is not a good thing to see, >>>> either... >>>> >>>> Please refer to https://lists.ipfire.org/pipermail/development/2020-February/007046.html >>>> for further details. >>>> >>>> Thanks, and best regards, >>>> Peter Müller >>>> >>>>> >>>>> >>>>>> On 31 Aug 2021, at 16:25, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> On 31.08.2021 11:56, Michael Tremer wrote: >>>>>>> Hey, >>>>>>> >>>>>>> This is an oops in the code that injects packets back into the kernel after they have been processed by suricata. >>>>>> >>>>>> Wow. How did you find this out!? >>>>> >>>>> There are two key functions in the trace: >>>>> >>>>>>>> 1 Time(s): nf_queue_entry_release_refs+0x82/0xa0 >>>>>>>> 1 Time(s): nf_reinject+0x7a/0x1e0 >>>>> >>>>> They are from Netfilter and the NFQUEUE module. We only use that for the IPS. >>>>> >>>>>>> Was this a one-off or does this happen on a regular basis? >>>>>> >>>>>> Until now, it only happened once. >>>>>> >>>>>> It rebooted the machine - just to be sure - and its running "without >>>>>> seen problems" since then. Absolutely normal. >>>>> >>>>> Would be interesting to see how it behaves if it doesn’t get a reboot. >>>>> >>>>> This is definitely a bug and needs to be fixed in the Linux kernel. >>>>> >>>>> -Michael >>>>> >>>>>> >>>>>> Best, >>>>>> Matthias >>>>>> >>>>>>> -Michael >>>>>>> >>>>>>>> On 27 Aug 2021, at 17:16, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> today I took the usual look in LOG SUMMARY and was surprised finding this: >>>>>>>> >>>>>>>> ***SNIP*** >>>>>>>> Kernel >>>>>>>> >>>>>>>> WARNING: Kernel Errors Present >>>>>>>> WARNING: CPU: 0 PID: 2984 at lib/refcount.c:28 r ...: 1 Time(s) >>>>>>>> >>>>>>>> 1 Time(s): ? nfnetlink_net_exit_batch+0x60/0x60 >>>>>>>> 1 Time(s): [last unloaded: hwmon_vid] >>>>>>>> 1 Time(s): ____sys_sendmsg+0x258/0x2a0 >>>>>>>> 1 Time(s): ___sys_sendmsg+0xa3/0xf0 >>>>>>>> 1 Time(s): __sys_sendmsg+0x81/0xd0 >>>>>>>> 1 Time(s): do_syscall_64+0x33/0x40 >>>>>>>> 1 Time(s): entry_SYSCALL_64_after_hwframe+0x44/0xa9 >>>>>>>> 1 Time(s): netlink_rcv_skb+0x5b/0x100 >>>>>>>> 1 Time(s): netlink_sendmsg+0x23a/0x470 >>>>>>>> 1 Time(s): netlink_unicast+0x209/0x2d0 >>>>>>>> 1 Time(s): nf_queue_entry_release_refs+0x82/0xa0 >>>>>>>> 1 Time(s): nf_reinject+0x7a/0x1e0 >>>>>>>> 1 Time(s): nfnetlink_rcv_msg+0x16d/0x2c0 >>>>>>>> 1 Time(s): nfqnl_recv_verdict+0x302/0x4f0 [nfnetlink_queue] >>>>>>>> 1 Time(s): sock_sendmsg+0x5e/0x60 >>>>>>>> 1 Time(s): ------------[ cut here ]------------ >>>>>>>> 1 Time(s): ---[ end trace 49e1e291edb98731 ]--- >>>>>>>> 1 Time(s): CPU: 0 PID: 2984 Comm: W-NFQ#1 Tainted: G O >>>>>>>> 5.10.55-ipfire #1 >>>>>>>> 1 Time(s): CR2: 000079fa1356a6f8 CR3: 0000000101532004 CR4: >>>>>>>> 00000000000706f0 >>>>>>>> 1 Time(s): CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>>>>>> 1 Time(s): Call Trace: >>>>>>>> 1 Time(s): Code: 05 ba 60 1f 01 01 e8 af a3 52 00 0f 0b c3 80 3d a8 60 >>>>>>>> 1f 01 00 75 95 48 c7 c7 b8 fc 11 b4 c6 05 98 60 1f 01 01 e8 90 a3 52 00 >>>>>>>> <0f> 0b c3 80 3d 87 60 1f 01 00 0f 85 72 ff ff ff 48 c7 c7 10 fd 11 >>>>>>>> 1 Time(s): Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 fa ee ff >>>>>>>> ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2e 00 00 00 0f 05 >>>>>>>> <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 4e ef ff ff 48 >>>>>>>> 1 Time(s): FS: 00007144d27d9640(0000) GS:ffff9db887200000(0000) >>>>>>>> knlGS:0000000000000000 >>>>>>>> 1 Time(s): Hardware name: To be filled by O.E.M. To be filled by >>>>>>>> O.E.M./CRESCENTBAY, BIOS 5.011 04/13/2016 >>>>>>>> 1 Time(s): Modules linked in: xt_REDIRECT nfnetlink_queue xt_NFQUEUE >>>>>>>> xt_MASQUERADE ccm cpufreq_conservative cpufreq_ondemand xt_geoip(O) >>>>>>>> xt_ipp2p(O) compat_xtables(O) xt_hashlimit xt_mac xt_multiport xt_mark >>>>>>>> xt_policy xt_TCPMSS xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 >>>>>>>> xt_LOG xt_limit nf_log_ipv4 nf_log_common iptable_raw iptable_mangle >>>>>>>> iptable_filter vfat fat sch_fq_codel rt2800usb rt2x00usb >>>>>>>> x86_pkg_temp_thermal rt2800lib intel_powerclamp coretemp rt2x00lib >>>>>>>> kvm_intel mac80211 kvm at24 regmap_i2c iTCO_wdt iTCO_vendor_support >>>>>>>> irqbypass crct10dif_pclmul cfg80211 crc32_pclmul ghash_clmulni_intel >>>>>>>> pcspkr i2c_i801 rfkill r8169 lpc_ich i2c_smbus mfd_core realtek libarc4 >>>>>>>> ir_rc6_decoder rc_rc6_mce i2c_algo_bit snd_hda_codec_realtek nuvoton_cir >>>>>>>> fb_sys_fops syscopyarea sysfillrect sysimgblt snd_hda_codec_generic >>>>>>>> ledtrig_audio i2c_core rc_core snd_hda_intel snd_intel_dspcfg >>>>>>>> snd_hda_codec snd_hda_core snd_hwdep acpi_pad snd_pcm snd_timer snd >>>>>>>> soundcore lp parport_pc parport video >>>>>>>> 1 Time(s): R10: 0000000000000000 R11: 0000000000000293 R12: >>>>>>>> 0000000000000000 >>>>>>>> 1 Time(s): R10: ffffb330821af780 R11: ffffffffb4533dc8 R12: >>>>>>>> ffff9db791371980 >>>>>>>> 1 Time(s): R13: 0000000000000000 R14: 0000000000000001 R15: >>>>>>>> 0000000000000070 >>>>>>>> 1 Time(s): R13: ffff9db7444d9200 R14: 0000000000000005 R15: >>>>>>>> ffff9db7f15c6b00 >>>>>>>> 1 Time(s): RAX: 0000000000000000 RBX: ffff9db791371980 RCX: >>>>>>>> 0000000000000027 >>>>>>>> 1 Time(s): RAX: ffffffffffffffda RBX: 00007144c4268dd0 RCX: >>>>>>>> 00007144d507062d >>>>>>>> 1 Time(s): RBP: 00007144d27d6fe0 R08: 0000000000000000 R09: >>>>>>>> 00007144d4754de0 >>>>>>>> 1 Time(s): RBP: ffffb330821afa50 R08: 0000000000000000 R09: >>>>>>>> ffffb330821af788 >>>>>>>> 1 Time(s): RDX: 0000000000000000 RSI: 00007144d27d6f80 RDI: >>>>>>>> 0000000000000006 >>>>>>>> 1 Time(s): RDX: ffff9db887218968 RSI: 0000000000000001 RDI: >>>>>>>> ffff9db887218960 >>>>>>>> 1 Time(s): RIP: 0010:refcount_warn_saturate+0xa6/0xf0 >>>>>>>> 1 Time(s): RIP: 0033:0x7144d507062d >>>>>>>> 1 Time(s): RSP: 0018:ffffb330821af950 EFLAGS: 00010282 >>>>>>>> 1 Time(s): RSP: 002b:00007144d27d6f40 EFLAGS: 00000293 ORIG_RAX: >>>>>>>> 000000000000002e >>>>>>>> 1 Time(s): refcount_t: underflow; use-after-free. >>>>>>>> ***SNAP*** >>>>>>>> >>>>>>>> I don't know exactly how do deal with this - does anyone has an idea >>>>>>>> what this means? >>>>>>>> >>>>>>>> Besides the machine is up and running with Core 159 / 64bit since 6 days >>>>>>>> now and if I hadn't looked at the logs I would not have noticed it. >>>>>>>> >>>>>>>> I didn't reboot or changed anything yet- should I do? >>>>>>>> >>>>>>>> Best, >>>>>>>> Matthias >>>>>>> >>>>>> >>>>> >>>> >>> >> > <kernel error - excerpt messages_02.txt> ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Core 159 - "Kernel errors present" in LOG SUMMARY 2021-11-28 17:03 ` Michael Tremer @ 2021-12-27 23:49 ` Matthias Fischer 0 siblings, 0 replies; 8+ messages in thread From: Matthias Fischer @ 2021-12-27 23:49 UTC (permalink / raw) To: development [-- Attachment #1: Type: text/plain, Size: 12876 bytes --] Hi, today - all of a sudden - it happened again (see attachment) with "IPFire 2.27 (x86_64) - core162". Profile: https://fireinfo.ipfire.org/profile/5f68a6360ffbecb6877dcac75f5b8c8030f43ce8 System was idle, nothing going on. But: where would be the right place to report this? Best, Matthias On 28.11.2021 18:03, Michael Tremer wrote: > Hello Matthias, > > Since we now have kernel 5.15.x in next, I would recommend installing that and check if the problem persists. > > Best, > -Michael > >> On 28 Nov 2021, at 14:21, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote: >> >> Hi, >> >> On 21.09.2021 11:36, Michael Tremer wrote: >>> Hello, >>> >>> Could we report this to the right place with the Linux kernel community? >> >> Where would be this "right place"? >> >> https://bugzilla.kernel.org/ or https://lkml.org/? >> >> It happened again today with Core 161 (running 'suricata 5.0.8'). >> >> => see Attachment >> >> Best, >> Matthias >> >>> This seems to be a long-standing problem which wouldn’t be very difficult to solve. >>> >>> -Michael >>> >>>> On 20 Sep 2021, at 16:16, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote: >>>> >>>> Hi, >>>> >>>> today it happened again - at 1:18am: >>>> >>>> ***SNIP*** >>>> ipfire kernel: refcount_t: underflow; use-after-free. >>>> ipfire kernel: WARNING: CPU: 1 PID: 30228 at lib/refcount.c:28 >>>> refcount_warn_saturate+0xa6/0xf0 >>>> ipfire kernel: Modules linked in: xt_REDIRECT nfnetlink_queue xt_NFQUEUE >>>> xt_MASQUERADE ccm cpufreq_conservative cpufreq_ondemand xt_geoip(O) >>>> xt_ipp2p(O) compat_xtables(O) xt_hashlimit xt_mac xt_multiport xt_mark >>>> xt_policy xt_TCPMSS xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 >>>> xt_LOG xt_limit nf_log_ipv4 nf_log_common iptable_raw iptable_mangle >>>> iptable_filter vfat fat rt2800usb rt2x00usb rt2800lib rt2x00lib >>>> sch_fq_codel x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel >>>> mac80211 at24 kvm regmap_i2c iTCO_wdt iTCO_vendor_support irqbypass >>>> crct10dif_pclmul crc32_pclmul cfg80211 ghash_clmulni_intel i2c_i801 >>>> pcspkr i2c_smbus lpc_ich rfkill mfd_core r8169 realtek libarc4 >>>> ir_rc6_decoder i2c_algo_bit fb_sys_fops snd_hda_codec_realtek >>>> syscopyarea rc_rc6_mce sysfillrect snd_hda_codec_generic sysimgblt >>>> nuvoton_cir i2c_core ledtrig_audio rc_core snd_hda_intel >>>> snd_intel_dspcfg snd_hda_codec snd_hda_core snd_hwdep acpi_pad snd_pcm >>>> snd_timer snd soundcore lp parport_pc parport video >>>> ipfire kernel: CPU: 1 PID: 30228 Comm: W-NFQ#0 Tainted: G O >>>> 5.10.55-ipfire #1 >>>> ipfire kernel: Hardware name: To be filled by O.E.M. To be filled by >>>> O.E.M./CRESCENTBAY, BIOS 5.011 04/13/2016 >>>> ipfire kernel: RIP: 0010:refcount_warn_saturate+0xa6/0xf0 >>>> ipfire kernel: Code: 05 ba 60 1f 01 01 e8 af a3 52 00 0f 0b c3 80 3d a8 >>>> 60 1f 01 00 75 95 48 c7 c7 b8 fc 11 ab c6 05 98 60 1f 01 01 e8 90 a3 52 >>>> 00 <0f> 0b c3 80 3d 87 60 1f 01 00 0f 85 72 ff ff ff 48 c7 c7 10 fd 11 >>>> ipfire kernel: RSP: 0018:ffffbaba82823950 EFLAGS: 00010282 >>>> ipfire kernel: RAX: 0000000000000000 RBX: ffff998e77a25500 RCX: >>>> 0000000000000027 >>>> ipfire kernel: RDX: ffff998f47318968 RSI: 0000000000000001 RDI: >>>> ffff998f47318960 >>>> ipfire kernel: RBP: ffffbaba82823a50 R08: 0000000000000000 R09: >>>> ffffbaba82823788 >>>> ipfire kernel: R10: ffffbaba82823780 R11: ffffffffab533dc8 R12: >>>> ffff998e77a25500 >>>> ipfire kernel: R13: ffff998e04193a00 R14: 0000000000000005 R15: >>>> ffff998e03a6ea00 >>>> ipfire kernel: FS: 0000758d85125640(0000) GS:ffff998f47300000(0000) >>>> knlGS:0000000000000000 >>>> ipfire kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>> ipfire kernel: CR2: 00007722aefed0a0 CR3: 000000010351e003 CR4: >>>> 00000000000706e0 >>>> ipfire kernel: Call Trace: >>>> ipfire kernel: nf_queue_entry_release_refs+0x82/0xa0 >>>> ipfire kernel: nf_reinject+0x7a/0x1e0 >>>> ipfire kernel: nfqnl_recv_verdict+0x302/0x4f0 [nfnetlink_queue] >>>> ipfire kernel: nfnetlink_rcv_msg+0x16d/0x2c0 >>>> ipfire kernel: ? nfnetlink_net_exit_batch+0x60/0x60 >>>> ipfire kernel: netlink_rcv_skb+0x5b/0x100 >>>> ipfire kernel: netlink_unicast+0x209/0x2d0 >>>> ipfire kernel: netlink_sendmsg+0x23a/0x470 >>>> ipfire kernel: sock_sendmsg+0x5e/0x60 >>>> ipfire kernel: ____sys_sendmsg+0x258/0x2a0 >>>> ipfire kernel: ___sys_sendmsg+0xa3/0xf0 >>>> ipfire kernel: __sys_sendmsg+0x81/0xd0 >>>> ipfire kernel: do_syscall_64+0x33/0x40 >>>> ipfire kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 >>>> ipfire kernel: RIP: 0033:0x758d871bb62d >>>> ipfire kernel: Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 fa ee >>>> ff ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2e 00 00 00 0f >>>> 05 <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 4e ef ff ff 48 >>>> ipfire kernel: RSP: 002b:0000758d85122f40 EFLAGS: 00000293 ORIG_RAX: >>>> 000000000000002e >>>> ipfire kernel: RAX: ffffffffffffffda RBX: 0000758d80268dd0 RCX: >>>> 0000758d871bb62d >>>> ipfire kernel: RDX: 0000000000000000 RSI: 0000758d85122f80 RDI: >>>> 0000000000000005 >>>> ipfire kernel: RBP: 0000758d85122fe0 R08: 0000000000000000 R09: >>>> 0000758d8689fde0 >>>> ipfire kernel: R10: 0000000000000000 R11: 0000000000000293 R12: >>>> 0000000000000000 >>>> ipfire kernel: R13: 0000000000000000 R14: 0000000000000001 R15: >>>> 0000000000000070 >>>> ipfire kernel: ---[ end trace 4c8c047c62e118e2 ]--- >>>> ***SNAP*** >>>> >>>> Machine is running without (seen) problems - I didn'T reboot yet. >>>> >>>> Best, >>>> Matthias >>>> >>>> On 09.09.2021 22:36, Peter Müller wrote: >>>>> Hello *, >>>>> >>>>> just for the records: I noticed this behaviour while testing Core Update 140/141 (in >>>>> February 2020) the first time. Not kept track on this, but it does not seem to cause >>>>> any harm - at least none I am aware of. But it certainly is not a good thing to see, >>>>> either... >>>>> >>>>> Please refer to https://lists.ipfire.org/pipermail/development/2020-February/007046.html >>>>> for further details. >>>>> >>>>> Thanks, and best regards, >>>>> Peter Müller >>>>> >>>>>> >>>>>> >>>>>>> On 31 Aug 2021, at 16:25, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> On 31.08.2021 11:56, Michael Tremer wrote: >>>>>>>> Hey, >>>>>>>> >>>>>>>> This is an oops in the code that injects packets back into the kernel after they have been processed by suricata. >>>>>>> >>>>>>> Wow. How did you find this out!? >>>>>> >>>>>> There are two key functions in the trace: >>>>>> >>>>>>>>> 1 Time(s): nf_queue_entry_release_refs+0x82/0xa0 >>>>>>>>> 1 Time(s): nf_reinject+0x7a/0x1e0 >>>>>> >>>>>> They are from Netfilter and the NFQUEUE module. We only use that for the IPS. >>>>>> >>>>>>>> Was this a one-off or does this happen on a regular basis? >>>>>>> >>>>>>> Until now, it only happened once. >>>>>>> >>>>>>> It rebooted the machine - just to be sure - and its running "without >>>>>>> seen problems" since then. Absolutely normal. >>>>>> >>>>>> Would be interesting to see how it behaves if it doesn’t get a reboot. >>>>>> >>>>>> This is definitely a bug and needs to be fixed in the Linux kernel. >>>>>> >>>>>> -Michael >>>>>> >>>>>>> >>>>>>> Best, >>>>>>> Matthias >>>>>>> >>>>>>>> -Michael >>>>>>>> >>>>>>>>> On 27 Aug 2021, at 17:16, Matthias Fischer <matthias.fischer(a)ipfire.org> wrote: >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> today I took the usual look in LOG SUMMARY and was surprised finding this: >>>>>>>>> >>>>>>>>> ***SNIP*** >>>>>>>>> Kernel >>>>>>>>> >>>>>>>>> WARNING: Kernel Errors Present >>>>>>>>> WARNING: CPU: 0 PID: 2984 at lib/refcount.c:28 r ...: 1 Time(s) >>>>>>>>> >>>>>>>>> 1 Time(s): ? nfnetlink_net_exit_batch+0x60/0x60 >>>>>>>>> 1 Time(s): [last unloaded: hwmon_vid] >>>>>>>>> 1 Time(s): ____sys_sendmsg+0x258/0x2a0 >>>>>>>>> 1 Time(s): ___sys_sendmsg+0xa3/0xf0 >>>>>>>>> 1 Time(s): __sys_sendmsg+0x81/0xd0 >>>>>>>>> 1 Time(s): do_syscall_64+0x33/0x40 >>>>>>>>> 1 Time(s): entry_SYSCALL_64_after_hwframe+0x44/0xa9 >>>>>>>>> 1 Time(s): netlink_rcv_skb+0x5b/0x100 >>>>>>>>> 1 Time(s): netlink_sendmsg+0x23a/0x470 >>>>>>>>> 1 Time(s): netlink_unicast+0x209/0x2d0 >>>>>>>>> 1 Time(s): nf_queue_entry_release_refs+0x82/0xa0 >>>>>>>>> 1 Time(s): nf_reinject+0x7a/0x1e0 >>>>>>>>> 1 Time(s): nfnetlink_rcv_msg+0x16d/0x2c0 >>>>>>>>> 1 Time(s): nfqnl_recv_verdict+0x302/0x4f0 [nfnetlink_queue] >>>>>>>>> 1 Time(s): sock_sendmsg+0x5e/0x60 >>>>>>>>> 1 Time(s): ------------[ cut here ]------------ >>>>>>>>> 1 Time(s): ---[ end trace 49e1e291edb98731 ]--- >>>>>>>>> 1 Time(s): CPU: 0 PID: 2984 Comm: W-NFQ#1 Tainted: G O >>>>>>>>> 5.10.55-ipfire #1 >>>>>>>>> 1 Time(s): CR2: 000079fa1356a6f8 CR3: 0000000101532004 CR4: >>>>>>>>> 00000000000706f0 >>>>>>>>> 1 Time(s): CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>>>>>>> 1 Time(s): Call Trace: >>>>>>>>> 1 Time(s): Code: 05 ba 60 1f 01 01 e8 af a3 52 00 0f 0b c3 80 3d a8 60 >>>>>>>>> 1f 01 00 75 95 48 c7 c7 b8 fc 11 b4 c6 05 98 60 1f 01 01 e8 90 a3 52 00 >>>>>>>>> <0f> 0b c3 80 3d 87 60 1f 01 00 0f 85 72 ff ff ff 48 c7 c7 10 fd 11 >>>>>>>>> 1 Time(s): Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 fa ee ff >>>>>>>>> ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2e 00 00 00 0f 05 >>>>>>>>> <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 4e ef ff ff 48 >>>>>>>>> 1 Time(s): FS: 00007144d27d9640(0000) GS:ffff9db887200000(0000) >>>>>>>>> knlGS:0000000000000000 >>>>>>>>> 1 Time(s): Hardware name: To be filled by O.E.M. To be filled by >>>>>>>>> O.E.M./CRESCENTBAY, BIOS 5.011 04/13/2016 >>>>>>>>> 1 Time(s): Modules linked in: xt_REDIRECT nfnetlink_queue xt_NFQUEUE >>>>>>>>> xt_MASQUERADE ccm cpufreq_conservative cpufreq_ondemand xt_geoip(O) >>>>>>>>> xt_ipp2p(O) compat_xtables(O) xt_hashlimit xt_mac xt_multiport xt_mark >>>>>>>>> xt_policy xt_TCPMSS xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 >>>>>>>>> xt_LOG xt_limit nf_log_ipv4 nf_log_common iptable_raw iptable_mangle >>>>>>>>> iptable_filter vfat fat sch_fq_codel rt2800usb rt2x00usb >>>>>>>>> x86_pkg_temp_thermal rt2800lib intel_powerclamp coretemp rt2x00lib >>>>>>>>> kvm_intel mac80211 kvm at24 regmap_i2c iTCO_wdt iTCO_vendor_support >>>>>>>>> irqbypass crct10dif_pclmul cfg80211 crc32_pclmul ghash_clmulni_intel >>>>>>>>> pcspkr i2c_i801 rfkill r8169 lpc_ich i2c_smbus mfd_core realtek libarc4 >>>>>>>>> ir_rc6_decoder rc_rc6_mce i2c_algo_bit snd_hda_codec_realtek nuvoton_cir >>>>>>>>> fb_sys_fops syscopyarea sysfillrect sysimgblt snd_hda_codec_generic >>>>>>>>> ledtrig_audio i2c_core rc_core snd_hda_intel snd_intel_dspcfg >>>>>>>>> snd_hda_codec snd_hda_core snd_hwdep acpi_pad snd_pcm snd_timer snd >>>>>>>>> soundcore lp parport_pc parport video >>>>>>>>> 1 Time(s): R10: 0000000000000000 R11: 0000000000000293 R12: >>>>>>>>> 0000000000000000 >>>>>>>>> 1 Time(s): R10: ffffb330821af780 R11: ffffffffb4533dc8 R12: >>>>>>>>> ffff9db791371980 >>>>>>>>> 1 Time(s): R13: 0000000000000000 R14: 0000000000000001 R15: >>>>>>>>> 0000000000000070 >>>>>>>>> 1 Time(s): R13: ffff9db7444d9200 R14: 0000000000000005 R15: >>>>>>>>> ffff9db7f15c6b00 >>>>>>>>> 1 Time(s): RAX: 0000000000000000 RBX: ffff9db791371980 RCX: >>>>>>>>> 0000000000000027 >>>>>>>>> 1 Time(s): RAX: ffffffffffffffda RBX: 00007144c4268dd0 RCX: >>>>>>>>> 00007144d507062d >>>>>>>>> 1 Time(s): RBP: 00007144d27d6fe0 R08: 0000000000000000 R09: >>>>>>>>> 00007144d4754de0 >>>>>>>>> 1 Time(s): RBP: ffffb330821afa50 R08: 0000000000000000 R09: >>>>>>>>> ffffb330821af788 >>>>>>>>> 1 Time(s): RDX: 0000000000000000 RSI: 00007144d27d6f80 RDI: >>>>>>>>> 0000000000000006 >>>>>>>>> 1 Time(s): RDX: ffff9db887218968 RSI: 0000000000000001 RDI: >>>>>>>>> ffff9db887218960 >>>>>>>>> 1 Time(s): RIP: 0010:refcount_warn_saturate+0xa6/0xf0 >>>>>>>>> 1 Time(s): RIP: 0033:0x7144d507062d >>>>>>>>> 1 Time(s): RSP: 0018:ffffb330821af950 EFLAGS: 00010282 >>>>>>>>> 1 Time(s): RSP: 002b:00007144d27d6f40 EFLAGS: 00000293 ORIG_RAX: >>>>>>>>> 000000000000002e >>>>>>>>> 1 Time(s): refcount_t: underflow; use-after-free. >>>>>>>>> ***SNAP*** >>>>>>>>> >>>>>>>>> I don't know exactly how do deal with this - does anyone has an idea >>>>>>>>> what this means? >>>>>>>>> >>>>>>>>> Besides the machine is up and running with Core 159 / 64bit since 6 days >>>>>>>>> now and if I hadn't looked at the logs I would not have noticed it. >>>>>>>>> >>>>>>>>> I didn't reboot or changed anything yet- should I do? >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Matthias >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> <kernel error - excerpt messages_02.txt> > [-- Attachment #2: kernelerror-excerptmessages_03.txt --] [-- Type: text/plain, Size: 5046 bytes --] Dec 27 23:08:30 ipfire kernel: ------------[ cut here ]------------ Dec 27 23:08:30 ipfire kernel: refcount_t: underflow; use-after-free. Dec 27 23:08:30 ipfire kernel: WARNING: CPU: 1 PID: 3026 at lib/refcount.c:28 refcount_warn_saturate+0xba/0x110 Dec 27 23:08:30 ipfire kernel: Modules linked in: xt_NFQUEUE nfnetlink_queue xt_MASQUERADE ccm cpufreq_conservative cpufreq_ondemand xt_geoip(O) xt_ipp2p(O) compat_xtables(O) xt_REDIRECT xt_hashlimit xt_mac xt_multiport xt_policy xt_TCPMSS xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 xt_LOG xt_limit xt_mark xt_connmark nf_log_syslog iptable_raw iptable_mangle iptable_filter vfat fat rt2800usb rt2x00usb x86_pkg_temp_thermal rt2800lib intel_powerclamp coretemp rt2x00lib mac80211 kvm_intel sch_fq_codel kvm snd_hda_codec_realtek snd_hda_codec_generic i2c_algo_bit fb_sys_fops ledtrig_audio syscopyarea at24 sysfillrect sysimgblt snd_hda_intel iTCO_wdt cfg80211 regmap_i2c iTCO_vendor_support snd_intel_dspcfg snd_hda_codec irqbypass crct10dif_pclmul i2c_i801 r8169 crc32_pclmul snd_hda_core i2c_smbus rfkill i2c_core ghash_clmulni_intel lpc_ich pcspkr realtek snd_hwdep libarc4 ir_rc6_decoder mfd_core rc_rc6_mce snd_pcm nuvoton_cir rc_core snd_timer snd soundcore acpi_pad lp parport_pc parport video efivarfs Dec 27 23:08:30 ipfire kernel: [last unloaded: hwmon_vid] Dec 27 23:08:30 ipfire kernel: CPU: 1 PID: 3026 Comm: W-NFQ#0 Tainted: G O 5.15.6-ipfire #1 Dec 27 23:08:30 ipfire kernel: Hardware name: To be filled by O.E.M. To be filled by O.E.M./CRESCENTBAY, BIOS 5.011 04/13/2016 Dec 27 23:08:30 ipfire kernel: RIP: 0010:refcount_warn_saturate+0xba/0x110 Dec 27 23:08:30 ipfire kernel: Code: 01 01 e8 32 91 59 00 0f 0b 31 f6 89 f7 c3 80 3d 14 8d 39 01 00 75 85 48 c7 c7 98 f6 73 8e c6 05 04 8d 39 01 01 e8 0f 91 59 00 <0f> 0b 31 f6 89 f7 c3 80 3d ef 8c 39 01 00 0f 85 5e ff ff ff 48 c7 Dec 27 23:08:30 ipfire kernel: RSP: 0018:ffffaf02025a7878 EFLAGS: 00010246 Dec 27 23:08:30 ipfire kernel: RAX: 0000000000000000 RBX: ffff88e4836a3980 RCX: 0000000000000000 Dec 27 23:08:30 ipfire kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 Dec 27 23:08:30 ipfire kernel: RBP: ffff88e4836a3980 R08: 0000000000000000 R09: 0000000000000000 Dec 27 23:08:30 ipfire kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff88e4836a3980 Dec 27 23:08:30 ipfire kernel: R13: 0000000000000000 R14: 0000000000000005 R15: ffff88e4afbbd200 Dec 27 23:08:30 ipfire kernel: FS: 00007d7409fc6640(0000) GS:ffff88e5c7300000(0000) knlGS:0000000000000000 Dec 27 23:08:30 ipfire kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Dec 27 23:08:30 ipfire kernel: CR2: 0000748ca3636868 CR3: 0000000103004006 CR4: 00000000000706e0 Dec 27 23:08:30 ipfire kernel: Call Trace: Dec 27 23:08:30 ipfire kernel: <TASK> Dec 27 23:08:30 ipfire kernel: nf_queue_entry_release_refs+0x8b/0xa0 Dec 27 23:08:30 ipfire kernel: nf_reinject+0x7a/0x1e0 Dec 27 23:08:30 ipfire kernel: nfqnl_recv_verdict+0x303/0x4f0 [nfnetlink_queue] Dec 27 23:08:30 ipfire kernel: nfnetlink_rcv_msg+0x251/0x310 Dec 27 23:08:30 ipfire kernel: ? nfnetlink_net_init+0xa0/0xa0 Dec 27 23:08:30 ipfire kernel: netlink_rcv_skb+0x5b/0x110 Dec 27 23:08:30 ipfire kernel: netlink_unicast+0x215/0x2e0 Dec 27 23:08:30 ipfire kernel: netlink_sendmsg+0x233/0x480 Dec 27 23:08:30 ipfire kernel: ? netlink_unicast+0x2e0/0x2e0 Dec 27 23:08:30 ipfire kernel: ____sys_sendmsg+0x2a6/0x2e0 Dec 27 23:08:30 ipfire kernel: ___sys_sendmsg+0xa3/0x100 Dec 27 23:08:30 ipfire kernel: __sys_sendmsg+0x81/0xe0 Dec 27 23:08:30 ipfire kernel: do_syscall_64+0x5c/0x90 Dec 27 23:08:30 ipfire kernel: ? syscall_exit_to_user_mode+0x23/0x50 Dec 27 23:08:30 ipfire kernel: ? __x64_sys_recvfrom+0x20/0x40 Dec 27 23:08:30 ipfire kernel: ? do_syscall_64+0x69/0x90 Dec 27 23:08:30 ipfire kernel: ? syscall_exit_to_user_mode+0x23/0x50 Dec 27 23:08:30 ipfire kernel: ? switch_fpu_return+0x51/0xc0 Dec 27 23:08:30 ipfire kernel: ? exit_to_user_mode_prepare+0x8d/0x160 Dec 27 23:08:30 ipfire kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae Dec 27 23:08:30 ipfire kernel: RIP: 0033:0x7d740c06162d Dec 27 23:08:30 ipfire kernel: Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 fa ee ff ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 4e ef ff ff 48 Dec 27 23:08:30 ipfire kernel: RSP: 002b:00007d7409fc3f40 EFLAGS: 00000293 ORIG_RAX: 000000000000002e Dec 27 23:08:30 ipfire kernel: RAX: ffffffffffffffda RBX: 00007d7404268dd0 RCX: 00007d740c06162d Dec 27 23:08:30 ipfire kernel: RDX: 0000000000000000 RSI: 00007d7409fc3f80 RDI: 0000000000000005 Dec 27 23:08:30 ipfire kernel: RBP: 00007d7409fc3fe0 R08: 0000000000000000 R09: 00007d740b740de0 Dec 27 23:08:30 ipfire kernel: R10: 00007d73fc57d340 R11: 0000000000000293 R12: 0000000000000000 Dec 27 23:08:30 ipfire kernel: R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000080 Dec 27 23:08:30 ipfire kernel: </TASK> Dec 27 23:08:30 ipfire kernel: ---[ end trace d19df8b93013567a ]--- ^ permalink raw reply [flat|nested] 8+ messages in thread
* Core 159 - "Kernel errors present" in LOG SUMMARY
@ 2021-08-27 16:16 Matthias Fischer
0 siblings, 0 replies; 8+ messages in thread
From: Matthias Fischer @ 2021-08-27 16:16 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 4180 bytes --]
Hi,
today I took the usual look in LOG SUMMARY and was surprised finding this:
***SNIP***
Kernel
WARNING: Kernel Errors Present
WARNING: CPU: 0 PID: 2984 at lib/refcount.c:28 r ...: 1 Time(s)
1 Time(s): ? nfnetlink_net_exit_batch+0x60/0x60
1 Time(s): [last unloaded: hwmon_vid]
1 Time(s): ____sys_sendmsg+0x258/0x2a0
1 Time(s): ___sys_sendmsg+0xa3/0xf0
1 Time(s): __sys_sendmsg+0x81/0xd0
1 Time(s): do_syscall_64+0x33/0x40
1 Time(s): entry_SYSCALL_64_after_hwframe+0x44/0xa9
1 Time(s): netlink_rcv_skb+0x5b/0x100
1 Time(s): netlink_sendmsg+0x23a/0x470
1 Time(s): netlink_unicast+0x209/0x2d0
1 Time(s): nf_queue_entry_release_refs+0x82/0xa0
1 Time(s): nf_reinject+0x7a/0x1e0
1 Time(s): nfnetlink_rcv_msg+0x16d/0x2c0
1 Time(s): nfqnl_recv_verdict+0x302/0x4f0 [nfnetlink_queue]
1 Time(s): sock_sendmsg+0x5e/0x60
1 Time(s): ------------[ cut here ]------------
1 Time(s): ---[ end trace 49e1e291edb98731 ]---
1 Time(s): CPU: 0 PID: 2984 Comm: W-NFQ#1 Tainted: G O
5.10.55-ipfire #1
1 Time(s): CR2: 000079fa1356a6f8 CR3: 0000000101532004 CR4:
00000000000706f0
1 Time(s): CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
1 Time(s): Call Trace:
1 Time(s): Code: 05 ba 60 1f 01 01 e8 af a3 52 00 0f 0b c3 80 3d a8 60
1f 01 00 75 95 48 c7 c7 b8 fc 11 b4 c6 05 98 60 1f 01 01 e8 90 a3 52 00
<0f> 0b c3 80 3d 87 60 1f 01 00 0f 85 72 ff ff ff 48 c7 c7 10 fd 11
1 Time(s): Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 fa ee ff
ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2e 00 00 00 0f 05
<48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 4e ef ff ff 48
1 Time(s): FS: 00007144d27d9640(0000) GS:ffff9db887200000(0000)
knlGS:0000000000000000
1 Time(s): Hardware name: To be filled by O.E.M. To be filled by
O.E.M./CRESCENTBAY, BIOS 5.011 04/13/2016
1 Time(s): Modules linked in: xt_REDIRECT nfnetlink_queue xt_NFQUEUE
xt_MASQUERADE ccm cpufreq_conservative cpufreq_ondemand xt_geoip(O)
xt_ipp2p(O) compat_xtables(O) xt_hashlimit xt_mac xt_multiport xt_mark
xt_policy xt_TCPMSS xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4
xt_LOG xt_limit nf_log_ipv4 nf_log_common iptable_raw iptable_mangle
iptable_filter vfat fat sch_fq_codel rt2800usb rt2x00usb
x86_pkg_temp_thermal rt2800lib intel_powerclamp coretemp rt2x00lib
kvm_intel mac80211 kvm at24 regmap_i2c iTCO_wdt iTCO_vendor_support
irqbypass crct10dif_pclmul cfg80211 crc32_pclmul ghash_clmulni_intel
pcspkr i2c_i801 rfkill r8169 lpc_ich i2c_smbus mfd_core realtek libarc4
ir_rc6_decoder rc_rc6_mce i2c_algo_bit snd_hda_codec_realtek nuvoton_cir
fb_sys_fops syscopyarea sysfillrect sysimgblt snd_hda_codec_generic
ledtrig_audio i2c_core rc_core snd_hda_intel snd_intel_dspcfg
snd_hda_codec snd_hda_core snd_hwdep acpi_pad snd_pcm snd_timer snd
soundcore lp parport_pc parport video
1 Time(s): R10: 0000000000000000 R11: 0000000000000293 R12:
0000000000000000
1 Time(s): R10: ffffb330821af780 R11: ffffffffb4533dc8 R12:
ffff9db791371980
1 Time(s): R13: 0000000000000000 R14: 0000000000000001 R15:
0000000000000070
1 Time(s): R13: ffff9db7444d9200 R14: 0000000000000005 R15:
ffff9db7f15c6b00
1 Time(s): RAX: 0000000000000000 RBX: ffff9db791371980 RCX:
0000000000000027
1 Time(s): RAX: ffffffffffffffda RBX: 00007144c4268dd0 RCX:
00007144d507062d
1 Time(s): RBP: 00007144d27d6fe0 R08: 0000000000000000 R09:
00007144d4754de0
1 Time(s): RBP: ffffb330821afa50 R08: 0000000000000000 R09:
ffffb330821af788
1 Time(s): RDX: 0000000000000000 RSI: 00007144d27d6f80 RDI:
0000000000000006
1 Time(s): RDX: ffff9db887218968 RSI: 0000000000000001 RDI:
ffff9db887218960
1 Time(s): RIP: 0010:refcount_warn_saturate+0xa6/0xf0
1 Time(s): RIP: 0033:0x7144d507062d
1 Time(s): RSP: 0018:ffffb330821af950 EFLAGS: 00010282
1 Time(s): RSP: 002b:00007144d27d6f40 EFLAGS: 00000293 ORIG_RAX:
000000000000002e
1 Time(s): refcount_t: underflow; use-after-free.
***SNAP***
I don't know exactly how do deal with this - does anyone has an idea
what this means?
Besides the machine is up and running with Core 159 / 64bit since 6 days
now and if I hadn't looked at the logs I would not have noticed it.
I didn't reboot or changed anything yet- should I do?
Best,
Matthias
^ permalink raw reply [flat|nested] 8+ messages in threadend of thread, other threads:[~2021-12-27 23:49 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-27 16:17 Core 159 - "Kernel errors present" in LOG SUMMARY Matthias Fischer
[not found] <F92EEFDA-65C9-45D6-8BCE-7ED5B6793313@ipfire.org>
2021-09-09 20:36 ` Peter Müller
2021-09-20 15:16 ` Matthias Fischer
2021-09-21 9:36 ` Michael Tremer
2021-11-28 14:21 ` Matthias Fischer
2021-11-28 17:03 ` Michael Tremer
2021-12-27 23:49 ` Matthias Fischer
-- strict thread matches above, loose matches on Subject: below --
2021-08-27 16:16 Matthias Fischer
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox