Re: Core Update 186 (testing) report

[Michael Tremer <michael.tremer@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 9883 bytes --]

Hello,

Just to update this thread I can only say that there must be some minor bug here. Searching through the recent commits in the kernel Git repository, there have not been any changes that I would obviously connect with this. So the most likely explanation is a rare race condition that might have been newly introduced or long existing. We don’t know.

That Suricata then starts running in a loop eating all the memory until it is finally killed is obviously a bad thing. So I suggest you report both problems to the respective upstreams and we see what we hear back from them. I don’t think that this should be a release blocker.

Best,
-Michael

> On 19 May 2024, at 16:37, Peter Müller <peter.mueller(a)ipfire.org> wrote:
> 
> Hello *,
> 
> I'm afraid I spoke too soon: Today, for unknown reasons, Suricata triggered the OOM killer:
> 
> May 19 11:49:26 maverick kernel: Suricata-Main invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
> May 19 11:49:26 maverick kernel: CPU: 3 PID: 5196 Comm: Suricata-Main Tainted: G      D            6.6.30-ipfire #1
> May 19 11:49:26 maverick kernel: [   5196]   101  5196   280087   115466  1634304    72864             0 Suricata-Main
> May 19 11:49:26 maverick kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=Suricata-Main,pid=5196,uid=101
> May 19 11:49:26 maverick kernel: Out of memory: Killed process 5196 (Suricata-Main) total-vm:1120348kB, anon-rss:461608kB, file-rss:256kB, shmem-rss:0kB, UID:101 pgtables:1596kB oom_score_adj:0
> 
> Attached to this e-mail is the memory consumption graph, which corroborates that, starting
> at around 10:40 AM, something was eating up an extraordinary amount of memory. The following
> might be related:
> 
> May 19 10:41:22 maverick kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000
> May 19 10:41:22 maverick kernel: #PF: supervisor instruction fetch in kernel mode
> May 19 10:41:22 maverick kernel: #PF: error_code(0x0010) - not-present page
> May 19 10:41:22 maverick kernel: PGD 0 P4D 0 
> May 19 10:41:22 maverick kernel: Oops: 0010 [#1] PREEMPT SMP PTI
> May 19 10:41:22 maverick kernel: CPU: 0 PID: 1585 Comm: tor Not tainted 6.6.30-ipfire #1
> May 19 10:41:22 maverick kernel: Hardware name: <redacted>
> May 19 10:41:22 maverick kernel: RIP: 0010:0x0
> May 19 10:41:22 maverick kernel: Code: Unable to access opcode bytes at 0xffffffffffffffd6.
> May 19 10:41:22 maverick kernel: RSP: 0018:ffffc90000433900 EFLAGS: 00010246
> May 19 10:41:22 maverick kernel: RAX: 0000000000000000 RBX: ffff88814cd371c0 RCX: 0000000000000000
> May 19 10:41:22 maverick kernel: RDX: 0000000000000000 RSI: ffffc900004339d8 RDI: 0000000000000000
> May 19 10:41:22 maverick kernel: RBP: 0000000000000218 R08: 0000000000000000 R09: 0000000000000000
> May 19 10:41:22 maverick kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000218
> May 19 10:41:22 maverick kernel: R13: ffff888101cf2d00 R14: 0000000000000040 R15: ffffc900004339d8
> May 19 10:41:22 maverick kernel: FS:  0000722d08168740(0000) GS:ffff88817bc00000(0000) knlGS:0000000000000000
> May 19 10:41:22 maverick kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> May 19 10:41:22 maverick kernel: CR2: ffffffffffffffd6 CR3: 000000010a9ee000 CR4: 00000000001006f0
> May 19 10:41:22 maverick kernel: Call Trace:
> May 19 10:41:22 maverick kernel:  <TASK>
> May 19 10:41:22 maverick kernel:  ? __die+0x23/0x80
> May 19 10:41:22 maverick kernel:  ? page_fault_oops+0x171/0x4e0
> May 19 10:41:22 maverick kernel:  ? nf_queue+0x18/0x50
> May 19 10:41:22 maverick kernel:  ? exc_page_fault+0x42c/0x730
> May 19 10:41:22 maverick kernel:  ? asm_exc_page_fault+0x26/0x30
> May 19 10:41:22 maverick kernel:  ? tcp_schedule_loss_probe+0x123/0x200
> May 19 10:41:22 maverick kernel:  ? tcp_write_xmit+0x1eb/0x1330
> May 19 10:41:22 maverick kernel:  ? tcp_sendmsg+0x2b/0x50
> May 19 10:41:22 maverick kernel:  ? sock_write_iter+0x15e/0x190
> May 19 10:41:22 maverick kernel:  ? vfs_write+0x3ab/0x450
> May 19 10:41:22 maverick kernel:  ? ksys_write+0xc3/0xf0
> May 19 10:41:22 maverick kernel:  ? do_syscall_64+0x5a/0x90
> May 19 10:41:22 maverick kernel:  ? exit_to_user_mode_prepare+0x1a/0x140
> May 19 10:41:22 maverick kernel:  ? syscall_exit_to_user_mode+0x2e/0x50
> May 19 10:41:22 maverick kernel:  ? do_syscall_64+0x66/0x90
> May 19 10:41:22 maverick kernel:  ? exit_to_user_mode_prepare+0x1a/0x140
> May 19 10:41:22 maverick kernel:  ? syscall_exit_to_user_mode+0x2e/0x50
> May 19 10:41:22 maverick kernel:  ? do_syscall_64+0x66/0x90
> May 19 10:41:22 maverick kernel:  ? vfs_write+0x3ab/0x450
> May 19 10:41:22 maverick kernel:  ? exit_to_user_mode_prepare+0x1a/0x140
> May 19 10:41:22 maverick kernel:  ? syscall_exit_to_user_mode+0x2e/0x50
> May 19 10:41:22 maverick kernel:  ? do_syscall_64+0x66/0x90
> May 19 10:41:22 maverick kernel:  ? exit_to_user_mode_prepare+0x1a/0x140
> May 19 10:41:22 maverick kernel:  ? syscall_exit_to_user_mode+0x2e/0x50
> May 19 10:41:22 maverick kernel:  ? do_syscall_64+0x66/0x90
> May 19 10:41:22 maverick kernel:  ? __hrtimer_run_queues+0x141/0x2b0
> May 19 10:41:22 maverick kernel:  ? __pfx_read_tsc+0x10/0x10
> May 19 10:41:22 maverick kernel:  ? ktime_get+0x43/0xb0
> May 19 10:41:22 maverick kernel:  ? lapic_next_deadline+0x2c/0x50
> May 19 10:41:22 maverick kernel:  ? clockevents_program_event+0x8d/0x100
> May 19 10:41:22 maverick kernel:  ? hrtimer_interrupt+0x12b/0x250
> May 19 10:41:22 maverick kernel:  ? exit_to_user_mode_prepare+0x1a/0x140
> May 19 10:41:22 maverick kernel:  ? entry_SYSCALL_64_after_hwframe+0x78/0xe2
> May 19 10:41:22 maverick kernel:  </TASK>
> May 19 10:41:22 maverick kernel: Modules linked in: esp4 tun act_mirred act_connmark em_ipt act_gact cls_basic ifb sch_ingress xt_layer7 cls_u32 sch_htb xt_NFQUEUE nfnetlink_queue xt_MASQUERADE pppoe pppox ppp_generic slhc 8021q garp xt_time xt_set ip_set_hash_net xt_REDIRECT xt_connlimit nf_conncount xt_multiport ip_set xt_owner xt_hashlimit xt_mac xt_policy xt_TCPMSS xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 xt_LOG xt_limit xt_mark xt_connmark nf_log_syslog iptable_raw iptable_mangle iptable_filter vfat fat snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio i915 ax88796b intel_rapl_common drm_buddy intel_powerclamp ttm coretemp drm_display_helper kvm_intel drm_kms_helper sch_cake i2c_algo_bit kvm snd_hda_intel snd_intel_dspcfg snd_hda_codec at24 iTCO_wdt regmap_i2c snd_hda_core iTCO_vendor_support asix mcs7830 snd_hwdep snd_pcm phylink usbnet snd_timer snd irqbypass mii i2c_i801 r8169 lpc_ich intel_xhci_usb_role_switch roles realtek i2c_smbus soundcore pcspkr mfd_core rfkill_gp
> May 19 10:41:22 maverick kernel: o rfkill
> May 19 10:41:22 maverick kernel:  intel_int0002_vgpio lp parport_pc parport efivarfs crct10dif_pclmul crc32_pclmul polyval_generic i2c_hid_acpi i2c_hid ghash_clmulni_intel sha512_ssse3 drm sha256_ssse3 sha1_ssse3 i2c_core video wmi dm_mirror dm_region_hash dm_log dm_mod btrfs blake2b_generic xor lzo_compress zstd_compress raid6_pq
> May 19 10:41:22 maverick kernel: CR2: 0000000000000000
> May 19 10:41:22 maverick kernel: ---[ end trace 0000000000000000 ]---
> May 19 10:41:22 maverick kernel: RIP: 0010:0x0
> May 19 10:41:22 maverick kernel: Code: Unable to access opcode bytes at 0xffffffffffffffd6.
> May 19 10:41:22 maverick kernel: RSP: 0018:ffffc90000433900 EFLAGS: 00010246
> May 19 10:41:22 maverick kernel: RAX: 0000000000000000 RBX: ffff88814cd371c0 RCX: 0000000000000000
> May 19 10:41:22 maverick kernel: RDX: 0000000000000000 RSI: ffffc900004339d8 RDI: 0000000000000000
> May 19 10:41:22 maverick kernel: RBP: 0000000000000218 R08: 0000000000000000 R09: 0000000000000000
> May 19 10:41:22 maverick kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000218
> May 19 10:41:22 maverick kernel: R13: ffff888101cf2d00 R14: 0000000000000040 R15: ffffc900004339d8
> May 19 10:41:22 maverick kernel: FS:  0000722d08168740(0000) GS:ffff88817bc00000(0000) knlGS:0000000000000000
> May 19 10:41:22 maverick kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> May 19 10:41:22 maverick kernel: CR2: ffffffffffffffd6 CR3: 000000010a9ee000 CR4: 00000000001006f0
> May 19 10:41:22 maverick kernel: note: tor[1585] exited with irqs disabled
> 
> I'm not sure what to make out of this, but it suggests that Core Update 186 needs a closer
> look before it is ready to be released.
> 
> Thanks, and best regards,
> Peter Müller
> 
>> Hello development folks,
>> 
>> Core Update 186 (testing; see: https://www.ipfire.org/blog/ipfire-2-29-core-update-186-is-available-for-testing)
>> is running here for a couple of days by now without any major issues known so far.
>> 
>> During the update, I merely noticed dracut complaining:
>> 
>>> dracut: Skipping program /bin/loginctl using in udev rule 71-seat.rules as it cannot be found
>> 
>> However, this does not appear to have any noticeable impact whatsoever.
>> 
>> The updated Lynis version now outputs significantly fewer warnings about deprecated
>> grep parameters, which previously made output hard to read sometimes.
>> 
>> Tested IPFire functionalities in detail:
>> - PPPoE dial-up via a DSL connection
>> - IPsec (N2N connections only)
>> - Squid (authentication enabled, using an upstream proxy)
>> - OpenVPN (RW connections only)
>> - IPS/Suricata (with Emerging Threats community ruleset enabled)
>> - Guardian
>> - Quality of Service
>> - DNS (using DNS over TLS and strict QNAME minimisation)
>> - Dynamic DNS
>> - Tor (relay mode)
>> 
>> I am looking forward to the release of Core Update 186.
>> 
>> Thanks, and best regards,
>> Peter Müller
> <getrrdimage.svg>