Re: Core Update 186 (testing) report

Re: Core Update 186 (testing) report

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 9883 bytes --]

Hello,

Just to update this thread I can only say that there must be some minor bug here. Searching through the recent commits in the kernel Git repository, there have not been any changes that I would obviously connect with this. So the most likely explanation is a rare race condition that might have been newly introduced or long existing. We don’t know.

That Suricata then starts running in a loop eating all the memory until it is finally killed is obviously a bad thing. So I suggest you report both problems to the respective upstreams and we see what we hear back from them. I don’t think that this should be a release blocker.

Best,
-Michael

> On 19 May 2024, at 16:37, Peter Müller <peter.mueller(a)ipfire.org> wrote:
> 
> Hello *,
> 
> I'm afraid I spoke too soon: Today, for unknown reasons, Suricata triggered the OOM killer:
> 
> May 19 11:49:26 maverick kernel: Suricata-Main invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
> May 19 11:49:26 maverick kernel: CPU: 3 PID: 5196 Comm: Suricata-Main Tainted: G      D            6.6.30-ipfire #1
> May 19 11:49:26 maverick kernel: [   5196]   101  5196   280087   115466  1634304    72864             0 Suricata-Main
> May 19 11:49:26 maverick kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=Suricata-Main,pid=5196,uid=101
> May 19 11:49:26 maverick kernel: Out of memory: Killed process 5196 (Suricata-Main) total-vm:1120348kB, anon-rss:461608kB, file-rss:256kB, shmem-rss:0kB, UID:101 pgtables:1596kB oom_score_adj:0
> 
> Attached to this e-mail is the memory consumption graph, which corroborates that, starting
> at around 10:40 AM, something was eating up an extraordinary amount of memory. The following
> might be related:
> 
> May 19 10:41:22 maverick kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000
> May 19 10:41:22 maverick kernel: #PF: supervisor instruction fetch in kernel mode
> May 19 10:41:22 maverick kernel: #PF: error_code(0x0010) - not-present page
> May 19 10:41:22 maverick kernel: PGD 0 P4D 0 
> May 19 10:41:22 maverick kernel: Oops: 0010 [#1] PREEMPT SMP PTI
> May 19 10:41:22 maverick kernel: CPU: 0 PID: 1585 Comm: tor Not tainted 6.6.30-ipfire #1
> May 19 10:41:22 maverick kernel: Hardware name: <redacted>
> May 19 10:41:22 maverick kernel: RIP: 0010:0x0
> May 19 10:41:22 maverick kernel: Code: Unable to access opcode bytes at 0xffffffffffffffd6.
> May 19 10:41:22 maverick kernel: RSP: 0018:ffffc90000433900 EFLAGS: 00010246
> May 19 10:41:22 maverick kernel: RAX: 0000000000000000 RBX: ffff88814cd371c0 RCX: 0000000000000000
> May 19 10:41:22 maverick kernel: RDX: 0000000000000000 RSI: ffffc900004339d8 RDI: 0000000000000000
> May 19 10:41:22 maverick kernel: RBP: 0000000000000218 R08: 0000000000000000 R09: 0000000000000000
> May 19 10:41:22 maverick kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000218
> May 19 10:41:22 maverick kernel: R13: ffff888101cf2d00 R14: 0000000000000040 R15: ffffc900004339d8
> May 19 10:41:22 maverick kernel: FS:  0000722d08168740(0000) GS:ffff88817bc00000(0000) knlGS:0000000000000000
> May 19 10:41:22 maverick kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> May 19 10:41:22 maverick kernel: CR2: ffffffffffffffd6 CR3: 000000010a9ee000 CR4: 00000000001006f0
> May 19 10:41:22 maverick kernel: Call Trace:
> May 19 10:41:22 maverick kernel:  <TASK>
> May 19 10:41:22 maverick kernel:  ? __die+0x23/0x80
> May 19 10:41:22 maverick kernel:  ? page_fault_oops+0x171/0x4e0
> May 19 10:41:22 maverick kernel:  ? nf_queue+0x18/0x50
> May 19 10:41:22 maverick kernel:  ? exc_page_fault+0x42c/0x730
> May 19 10:41:22 maverick kernel:  ? asm_exc_page_fault+0x26/0x30
> May 19 10:41:22 maverick kernel:  ? tcp_schedule_loss_probe+0x123/0x200
> May 19 10:41:22 maverick kernel:  ? tcp_write_xmit+0x1eb/0x1330
> May 19 10:41:22 maverick kernel:  ? tcp_sendmsg+0x2b/0x50
> May 19 10:41:22 maverick kernel:  ? sock_write_iter+0x15e/0x190
> May 19 10:41:22 maverick kernel:  ? vfs_write+0x3ab/0x450
> May 19 10:41:22 maverick kernel:  ? ksys_write+0xc3/0xf0
> May 19 10:41:22 maverick kernel:  ? do_syscall_64+0x5a/0x90
> May 19 10:41:22 maverick kernel:  ? exit_to_user_mode_prepare+0x1a/0x140
> May 19 10:41:22 maverick kernel:  ? syscall_exit_to_user_mode+0x2e/0x50
> May 19 10:41:22 maverick kernel:  ? do_syscall_64+0x66/0x90
> May 19 10:41:22 maverick kernel:  ? exit_to_user_mode_prepare+0x1a/0x140
> May 19 10:41:22 maverick kernel:  ? syscall_exit_to_user_mode+0x2e/0x50
> May 19 10:41:22 maverick kernel:  ? do_syscall_64+0x66/0x90
> May 19 10:41:22 maverick kernel:  ? vfs_write+0x3ab/0x450
> May 19 10:41:22 maverick kernel:  ? exit_to_user_mode_prepare+0x1a/0x140
> May 19 10:41:22 maverick kernel:  ? syscall_exit_to_user_mode+0x2e/0x50
> May 19 10:41:22 maverick kernel:  ? do_syscall_64+0x66/0x90
> May 19 10:41:22 maverick kernel:  ? exit_to_user_mode_prepare+0x1a/0x140
> May 19 10:41:22 maverick kernel:  ? syscall_exit_to_user_mode+0x2e/0x50
> May 19 10:41:22 maverick kernel:  ? do_syscall_64+0x66/0x90
> May 19 10:41:22 maverick kernel:  ? __hrtimer_run_queues+0x141/0x2b0
> May 19 10:41:22 maverick kernel:  ? __pfx_read_tsc+0x10/0x10
> May 19 10:41:22 maverick kernel:  ? ktime_get+0x43/0xb0
> May 19 10:41:22 maverick kernel:  ? lapic_next_deadline+0x2c/0x50
> May 19 10:41:22 maverick kernel:  ? clockevents_program_event+0x8d/0x100
> May 19 10:41:22 maverick kernel:  ? hrtimer_interrupt+0x12b/0x250
> May 19 10:41:22 maverick kernel:  ? exit_to_user_mode_prepare+0x1a/0x140
> May 19 10:41:22 maverick kernel:  ? entry_SYSCALL_64_after_hwframe+0x78/0xe2
> May 19 10:41:22 maverick kernel:  </TASK>
> May 19 10:41:22 maverick kernel: Modules linked in: esp4 tun act_mirred act_connmark em_ipt act_gact cls_basic ifb sch_ingress xt_layer7 cls_u32 sch_htb xt_NFQUEUE nfnetlink_queue xt_MASQUERADE pppoe pppox ppp_generic slhc 8021q garp xt_time xt_set ip_set_hash_net xt_REDIRECT xt_connlimit nf_conncount xt_multiport ip_set xt_owner xt_hashlimit xt_mac xt_policy xt_TCPMSS xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 xt_LOG xt_limit xt_mark xt_connmark nf_log_syslog iptable_raw iptable_mangle iptable_filter vfat fat snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio i915 ax88796b intel_rapl_common drm_buddy intel_powerclamp ttm coretemp drm_display_helper kvm_intel drm_kms_helper sch_cake i2c_algo_bit kvm snd_hda_intel snd_intel_dspcfg snd_hda_codec at24 iTCO_wdt regmap_i2c snd_hda_core iTCO_vendor_support asix mcs7830 snd_hwdep snd_pcm phylink usbnet snd_timer snd irqbypass mii i2c_i801 r8169 lpc_ich intel_xhci_usb_role_switch roles realtek i2c_smbus soundcore pcspkr mfd_core rfkill_gp
> May 19 10:41:22 maverick kernel: o rfkill
> May 19 10:41:22 maverick kernel:  intel_int0002_vgpio lp parport_pc parport efivarfs crct10dif_pclmul crc32_pclmul polyval_generic i2c_hid_acpi i2c_hid ghash_clmulni_intel sha512_ssse3 drm sha256_ssse3 sha1_ssse3 i2c_core video wmi dm_mirror dm_region_hash dm_log dm_mod btrfs blake2b_generic xor lzo_compress zstd_compress raid6_pq
> May 19 10:41:22 maverick kernel: CR2: 0000000000000000
> May 19 10:41:22 maverick kernel: ---[ end trace 0000000000000000 ]---
> May 19 10:41:22 maverick kernel: RIP: 0010:0x0
> May 19 10:41:22 maverick kernel: Code: Unable to access opcode bytes at 0xffffffffffffffd6.
> May 19 10:41:22 maverick kernel: RSP: 0018:ffffc90000433900 EFLAGS: 00010246
> May 19 10:41:22 maverick kernel: RAX: 0000000000000000 RBX: ffff88814cd371c0 RCX: 0000000000000000
> May 19 10:41:22 maverick kernel: RDX: 0000000000000000 RSI: ffffc900004339d8 RDI: 0000000000000000
> May 19 10:41:22 maverick kernel: RBP: 0000000000000218 R08: 0000000000000000 R09: 0000000000000000
> May 19 10:41:22 maverick kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000218
> May 19 10:41:22 maverick kernel: R13: ffff888101cf2d00 R14: 0000000000000040 R15: ffffc900004339d8
> May 19 10:41:22 maverick kernel: FS:  0000722d08168740(0000) GS:ffff88817bc00000(0000) knlGS:0000000000000000
> May 19 10:41:22 maverick kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> May 19 10:41:22 maverick kernel: CR2: ffffffffffffffd6 CR3: 000000010a9ee000 CR4: 00000000001006f0
> May 19 10:41:22 maverick kernel: note: tor[1585] exited with irqs disabled
> 
> I'm not sure what to make out of this, but it suggests that Core Update 186 needs a closer
> look before it is ready to be released.
> 
> Thanks, and best regards,
> Peter Müller
> 
>> Hello development folks,
>> 
>> Core Update 186 (testing; see: https://www.ipfire.org/blog/ipfire-2-29-core-update-186-is-available-for-testing)
>> is running here for a couple of days by now without any major issues known so far.
>> 
>> During the update, I merely noticed dracut complaining:
>> 
>>> dracut: Skipping program /bin/loginctl using in udev rule 71-seat.rules as it cannot be found
>> 
>> However, this does not appear to have any noticeable impact whatsoever.
>> 
>> The updated Lynis version now outputs significantly fewer warnings about deprecated
>> grep parameters, which previously made output hard to read sometimes.
>> 
>> Tested IPFire functionalities in detail:
>> - PPPoE dial-up via a DSL connection
>> - IPsec (N2N connections only)
>> - Squid (authentication enabled, using an upstream proxy)
>> - OpenVPN (RW connections only)
>> - IPS/Suricata (with Emerging Threats community ruleset enabled)
>> - Guardian
>> - Quality of Service
>> - DNS (using DNS over TLS and strict QNAME minimisation)
>> - Dynamic DNS
>> - Tor (relay mode)
>> 
>> I am looking forward to the release of Core Update 186.
>> 
>> Thanks, and best regards,
>> Peter Müller
> <getrrdimage.svg>

Core Update 186 (testing) report

[Peter Müller]

[-- Attachment #1: Type: text/plain, Size: 1121 bytes --]

Hello development folks,

Core Update 186 (testing; see: https://www.ipfire.org/blog/ipfire-2-29-core-update-186-is-available-for-testing)
is running here for a couple of days by now without any major issues known so far.

During the update, I merely noticed dracut complaining:

> dracut: Skipping program /bin/loginctl using in udev rule 71-seat.rules as it cannot be found

However, this does not appear to have any noticeable impact whatsoever.

The updated Lynis version now outputs significantly fewer warnings about deprecated
grep parameters, which previously made output hard to read sometimes.

Tested IPFire functionalities in detail:
- PPPoE dial-up via a DSL connection
- IPsec (N2N connections only)
- Squid (authentication enabled, using an upstream proxy)
- OpenVPN (RW connections only)
- IPS/Suricata (with Emerging Threats community ruleset enabled)
- Guardian
- Quality of Service
- DNS (using DNS over TLS and strict QNAME minimisation)
- Dynamic DNS
- Tor (relay mode)

I am looking forward to the release of Core Update 186.

Thanks, and best regards,
Peter Müller