Im not sure but I have found the same trace on my sytem on 1. September (without the patch) so it should not caused by the patch itself. I have also put packages to install patched kernel to my people for better testing. (i586 currently missing, helena still build it...) https://people.ipfire.org/~arne_f/highly-experimental/kernel/ Arne Am 2019-09-24 00:17, schrieb Michael Tremer: > What are we doing about these kernel traces? > > They don’t look like we can ignore them. > > -Michael > >> On 19 Sep 2019, at 21:16, peter.mueller(a)ipfire.org wrote: >> >> Hello Stefan, >> >> sorry for my late reply on this topic. Finally, I was able to >> test your new ISO, next-suricata-rust/d22217e1-dirty . >> >> New connections, regardless of SSH, TLS, DNS or whatever, are >> now established instantly. No packets are dropped anymore, this >> issue seems to be solved by the changes included in your ISO. >> >> OpenVPN RW performance, however, is worse than before: Now around >> ~ 400 kB/sec. if Suricata is enabled, and somewhere around 900 kB/sec. >> if it is not. I am getting the feeling this is an OpenVPN bug or >> performance issue, but as mentioned several times before, I am not >> satisfied with OpenVPN anyway. >> >> A few minutes after booting, kernel emits these log lines: >>> Sep 19 22:01:51 maverick kernel: refcount_t: increment on 0; >>> use-after-free. >>> Sep 19 22:01:51 maverick kernel: ------------[ cut here ]------------ >>> Sep 19 22:01:51 maverick kernel: WARNING: CPU: 2 PID: 2510 at >>> lib/refcount.c:153 refcount_inc.cold.12+0x13/0x16 >>> Sep 19 22:01:51 maverick kernel: Modules linked in: xt_IMQ imq >>> xt_length xt_DSCP xt_layer7 cls_fw sch_htb chacha20_x86_64 >>> chacha20_generic poly1305_x86_64 poly1305_generic chacha20poly1305 >>> esp4 xfrm6_mode_tunnel xfrm4_mode_tunnel tun nfnetlink_queue >>> xt_NFQUEUE ipt_MASQUERADE nf_nat_masquerade_ipv4 pppoe pppox >>> ppp_generic slhc xt_geoip(O) xt_connlimit xt_multiport xt_hashlimit >>> xt_mark xt_policy xt_TCPMSS nf_nat_irc nf_conntrack_irc nf_nat_tftp >>> nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp xt_CT xt_helper >>> nf_nat_h323 nf_conntrack_h323 xt_conntrack xt_comment ipt_REJECT >>> nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG xt_limit iptable_raw >>> iptable_mangle iptable_filter vfat fat snd_hda_codec_hdmi >>> sch_fq_codel snd_hda_codec_realtek snd_hda_codec_generic >>> intel_powerclamp coretemp kvm_intel i2c_algo_bit fb_sys_fops >>> syscopyarea >>> Sep 19 22:01:51 maverick kernel: sysfillrect sysimgblt snd_hda_intel >>> kvm snd_hda_codec snd_hda_core iTCO_wdt iTCO_vendor_support irqbypass >>> snd_hwdep crct10dif_pclmul mcs7830 crc32_pclmul snd_pcm >>> ghash_clmulni_intel snd_timer usbnet r8169 pcspkr snd i2c_i801 >>> lpc_ich mfd_core mii soundcore i2c_hid i2c_core pcc_cpufreq >>> rfkill_gpio rfkill intel_int0002_vgpio lp parport_pc parport video >>> Sep 19 22:01:51 maverick kernel: CPU: 2 PID: 2510 Comm: W-Q3 Tainted: >>> G O 4.14.138-ipfire #1 >>> Sep 19 22:01:51 maverick kernel: Hardware name: Gigabyte Technology >>> Co., Ltd. Default string/N3150ND3V, BIOS F5a 01/19/2018 >>> Sep 19 22:01:51 maverick kernel: task: ffffa393374fb200 task.stack: >>> ffffa98f00290000 >>> Sep 19 22:01:51 maverick kernel: RIP: >>> 0010:refcount_inc.cold.12+0x13/0x16 >>> Sep 19 22:01:51 maverick kernel: RSP: 0018:ffffa98f00293798 EFLAGS: >>> 00010246 >>> Sep 19 22:01:51 maverick kernel: RAX: 000000000000002b RBX: >>> ffffa39338e7ed00 RCX: 0000000000000000 >>> Sep 19 22:01:51 maverick kernel: RDX: 0000000000000000 RSI: >>> ffffa3933fd163f8 RDI: ffffa3933fd163f8 >>> Sep 19 22:01:51 maverick kernel: RBP: ffffffffb3e9e220 R08: >>> 000000000000003c R09: 000000000000027c >>> Sep 19 22:01:51 maverick kernel: R10: 0000000000000000 R11: >>> 0000000000000001 R12: ffffa39338c81100 >>> Sep 19 22:01:51 maverick kernel: R13: ffffffffb44c35c0 R14: >>> 0000000000028003 R15: ffffffffc03fc3e0 >>> Sep 19 22:01:51 maverick kernel: FS: 00007309737fe700(0000) >>> GS:ffffa3933fd00000(0000) knlGS:0000000000000000 >>> Sep 19 22:01:51 maverick kernel: CS: 0010 DS: 0000 ES: 0000 CR0: >>> 0000000080050033 >>> Sep 19 22:01:51 maverick kernel: CR2: 00007eb62b581728 CR3: >>> 0000000177e30000 CR4: 00000000001006e0 >>> Sep 19 22:01:51 maverick kernel: Call Trace: >>> Sep 19 22:01:51 maverick kernel: nf_queue_entry_get_refs+0x41/0x90 >>> Sep 19 22:01:51 maverick kernel: nf_queue+0xf9/0x220 >>> Sep 19 22:01:51 maverick kernel: nf_hook_slow+0x9f/0xf0 >>> Sep 19 22:01:51 maverick kernel: __ip_local_out+0xe4/0x150 >>> Sep 19 22:01:51 maverick kernel: ? >>> ip_forward_options.cold.7+0x27/0x27 >>> Sep 19 22:01:51 maverick kernel: xfrm_output_resume+0x21e/0x540 >>> Sep 19 22:01:51 maverick kernel: ? ipv4_confirm+0x3f/0xd0 >>> Sep 19 22:01:51 maverick kernel: xfrm4_output+0x3a/0xe0 >>> Sep 19 22:01:51 maverick kernel: ? xfrm4_udp_encap_rcv+0x1a0/0x1a0 >>> Sep 19 22:01:51 maverick kernel: nf_reinject+0x176/0x190 >>> Sep 19 22:01:51 maverick kernel: nfqnl_recv_verdict+0x293/0x4a0 >>> [nfnetlink_queue] >>> Sep 19 22:01:51 maverick kernel: ? nla_parse+0xb5/0xe0 >>> Sep 19 22:01:51 maverick kernel: nfnetlink_rcv_msg+0x14e/0x260 >>> Sep 19 22:01:51 maverick kernel: ? >>> nfnetlink_net_exit_batch+0x60/0x60 >>> Sep 19 22:01:51 maverick kernel: netlink_rcv_skb+0x78/0x150 >>> Sep 19 22:01:51 maverick kernel: nfnetlink_rcv+0x70/0x760 >>> Sep 19 22:01:51 maverick kernel: ? lock_timer_base+0x67/0x80 >>> Sep 19 22:01:51 maverick kernel: ? try_to_del_timer_sync+0x4d/0x80 >>> Sep 19 22:01:51 maverick kernel: ? __netlink_lookup+0xe1/0x140 >>> Sep 19 22:01:51 maverick kernel: netlink_unicast+0x183/0x230 >>> Sep 19 22:01:51 maverick kernel: netlink_sendmsg+0x204/0x3d0 >>> Sep 19 22:01:51 maverick kernel: sock_sendmsg+0x36/0x40 >>> Sep 19 22:01:51 maverick kernel: ___sys_sendmsg+0x2a7/0x300 >>> Sep 19 22:01:51 maverick kernel: ? netlink_recvmsg+0x398/0x460 >>> Sep 19 22:01:51 maverick kernel: ? __switch_to_asm+0x41/0x70 >>> Sep 19 22:01:51 maverick kernel: __sys_sendmsg+0x67/0xb0 >>> Sep 19 22:01:51 maverick kernel: do_syscall_64+0x67/0x100 >>> Sep 19 22:01:51 maverick kernel: >>> entry_SYSCALL_64_after_hwframe+0x3d/0xa2 >>> Sep 19 22:01:51 maverick kernel: RIP: 0033:0x73097af4a8c7 >>> Sep 19 22:01:51 maverick kernel: RSP: 002b:00007309737fbfc0 EFLAGS: >>> 00000293 ORIG_RAX: 000000000000002e >>> Sep 19 22:01:51 maverick kernel: RAX: ffffffffffffffda RBX: >>> 0000000000000008 RCX: 000073097af4a8c7 >>> Sep 19 22:01:51 maverick kernel: RDX: 0000000000000000 RSI: >>> 00007309737fc000 RDI: 0000000000000008 >>> Sep 19 22:01:51 maverick kernel: RBP: 00007309737fc000 R08: >>> 0000000000000000 R09: 0000000000000301 >>> Sep 19 22:01:51 maverick kernel: R10: 0000730974409e20 R11: >>> 0000000000000293 R12: 0000000000000000 >>> Sep 19 22:01:51 maverick kernel: R13: 000073096026cd98 R14: >>> 0000000000000070 R15: 0000000000000001 >>> Sep 19 22:01:51 maverick kernel: Code: 08 90 67 b4 c6 05 5e ca d8 00 >>> 01 e8 f7 e5 d7 ff 0f 0b b8 01 00 00 00 c3 48 c7 c7 60 90 67 b4 c6 05 >>> 42 ca d8 00 01 e8 dc e5 d7 ff <0f> 0b c3 48 c7 c7 90 90 67 b4 c6 05 >>> 2b ca d8 00 01 e8 c6 e5 d7 >>> Sep 19 22:01:51 maverick kernel: ---[ end trace dc2e33bbb9167d28 ]--- >>> Sep 19 22:01:51 maverick kernel: refcount_t: underflow; >>> use-after-free. >>> Sep 19 22:01:51 maverick kernel: ------------[ cut here ]------------ >>> Sep 19 22:01:51 maverick kernel: WARNING: CPU: 3 PID: 2507 at >>> lib/refcount.c:187 refcount_sub_and_test.cold.13+0x13/0x1a >>> Sep 19 22:01:51 maverick kernel: Modules linked in: xt_IMQ imq >>> xt_length xt_DSCP xt_layer7 cls_fw sch_htb chacha20_x86_64 >>> chacha20_generic poly1305_x86_64 poly1305_generic chacha20poly1305 >>> esp4 xfrm6_mode_tunnel xfrm4_mode_tunnel tun nfnetlink_queue >>> xt_NFQUEUE ipt_MASQUERADE nf_nat_masquerade_ipv4 pppoe pppox >>> ppp_generic slhc xt_geoip(O) xt_connlimit xt_multiport xt_hashlimit >>> xt_mark xt_policy xt_TCPMSS nf_nat_irc nf_conntrack_irc nf_nat_tftp >>> nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp xt_CT xt_helper >>> nf_nat_h323 nf_conntrack_h323 xt_conntrack xt_comment ipt_REJECT >>> nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG xt_limit iptable_raw >>> iptable_mangle iptable_filter vfat fat snd_hda_codec_hdmi >>> sch_fq_codel snd_hda_codec_realtek snd_hda_codec_generic >>> intel_powerclamp coretemp kvm_intel i2c_algo_bit fb_sys_fops >>> syscopyarea >>> Sep 19 22:01:51 maverick kernel: sysfillrect sysimgblt snd_hda_intel >>> kvm snd_hda_codec snd_hda_core iTCO_wdt iTCO_vendor_support irqbypass >>> snd_hwdep crct10dif_pclmul mcs7830 crc32_pclmul snd_pcm >>> ghash_clmulni_intel snd_timer usbnet r8169 pcspkr snd i2c_i801 >>> lpc_ich mfd_core mii soundcore i2c_hid i2c_core pcc_cpufreq >>> rfkill_gpio rfkill intel_int0002_vgpio lp parport_pc parport video >>> Sep 19 22:01:51 maverick kernel: CPU: 3 PID: 2507 Comm: W-Q2 Tainted: >>> G W O 4.14.138-ipfire #1 >>> Sep 19 22:01:51 maverick kernel: Hardware name: Gigabyte Technology >>> Co., Ltd. Default string/N3150ND3V, BIOS F5a 01/19/2018 >>> Sep 19 22:01:51 maverick kernel: task: ffffa393395ea580 task.stack: >>> ffffa98f00250000 >>> Sep 19 22:01:51 maverick kernel: RIP: >>> 0010:refcount_sub_and_test.cold.13+0x13/0x1a >>> Sep 19 22:01:51 maverick kernel: RSP: 0018:ffffa98f00253928 EFLAGS: >>> 00010246 >>> Sep 19 22:01:51 maverick kernel: RAX: 0000000000000026 RBX: >>> 0000000000000000 RCX: 0000000000000000 >>> Sep 19 22:01:51 maverick kernel: RDX: 0000000000000000 RSI: >>> ffffa3933fd963f8 RDI: ffffa3933fd963f8 >>> Sep 19 22:01:51 maverick kernel: RBP: ffffa39339bc4400 R08: >>> 0000000000000038 R09: 00000000000002b4 >>> Sep 19 22:01:51 maverick kernel: R10: 0000000000000000 R11: >>> 0000000000000001 R12: ffffa393383ab480 >>> Sep 19 22:01:51 maverick kernel: R13: ffffa39337f5da00 R14: >>> ffffa39339cd9840 R15: 0000000000000000 >>> Sep 19 22:01:51 maverick kernel: FS: 0000730973fff700(0000) >>> GS:ffffa3933fd80000(0000) knlGS:0000000000000000 >>> Sep 19 22:01:51 maverick kernel: CS: 0010 DS: 0000 ES: 0000 CR0: >>> 0000000080050033 >>> Sep 19 22:01:51 maverick kernel: CR2: 0000000002218828 CR3: >>> 0000000177e30000 CR4: 00000000001006e0 >>> Sep 19 22:01:51 maverick kernel: Call Trace: >>> Sep 19 22:01:51 maverick kernel: >>> nf_queue_entry_release_refs+0x45/0xa0 >>> Sep 19 22:01:51 maverick kernel: nf_reinject+0x3d/0x190 >>> Sep 19 22:01:51 maverick kernel: nfqnl_recv_verdict+0x293/0x4a0 >>> [nfnetlink_queue] >>> Sep 19 22:01:51 maverick kernel: ? nla_parse+0xb5/0xe0 >>> Sep 19 22:01:51 maverick kernel: nfnetlink_rcv_msg+0x14e/0x260 >>> Sep 19 22:01:51 maverick kernel: ? >>> nfnetlink_net_exit_batch+0x60/0x60 >>> Sep 19 22:01:51 maverick kernel: netlink_rcv_skb+0x78/0x150 >>> Sep 19 22:01:51 maverick kernel: nfnetlink_rcv+0x70/0x760 >>> Sep 19 22:01:51 maverick kernel: ? lock_timer_base+0x67/0x80 >>> Sep 19 22:01:51 maverick kernel: ? try_to_del_timer_sync+0x4d/0x80 >>> Sep 19 22:01:51 maverick kernel: ? __netlink_lookup+0xe1/0x140 >>> Sep 19 22:01:51 maverick kernel: netlink_unicast+0x183/0x230 >>> Sep 19 22:01:51 maverick kernel: netlink_sendmsg+0x204/0x3d0 >>> Sep 19 22:01:51 maverick kernel: sock_sendmsg+0x36/0x40 >>> Sep 19 22:01:51 maverick kernel: ___sys_sendmsg+0x2a7/0x300 >>> Sep 19 22:01:51 maverick kernel: ? netlink_recvmsg+0x398/0x460 >>> Sep 19 22:01:51 maverick kernel: ? __switch_to_asm+0x41/0x70 >>> Sep 19 22:01:51 maverick kernel: __sys_sendmsg+0x67/0xb0 >>> Sep 19 22:01:51 maverick kernel: do_syscall_64+0x67/0x100 >>> Sep 19 22:01:51 maverick kernel: >>> entry_SYSCALL_64_after_hwframe+0x3d/0xa2 >>> Sep 19 22:01:51 maverick kernel: RIP: 0033:0x73097af4a8c7 >>> Sep 19 22:01:51 maverick kernel: RSP: 002b:0000730973ffcfc0 EFLAGS: >>> 00000293 ORIG_RAX: 000000000000002e >>> Sep 19 22:01:51 maverick kernel: RAX: ffffffffffffffda RBX: >>> 0000000000000007 RCX: 000073097af4a8c7 >>> Sep 19 22:01:51 maverick kernel: RDX: 0000000000000000 RSI: >>> 0000730973ffd000 RDI: 0000000000000007 >>> Sep 19 22:01:51 maverick kernel: RBP: 0000730973ffd000 R08: >>> 0000000000000000 R09: 0000000000000301 >>> Sep 19 22:01:51 maverick kernel: R10: 000073096826c188 R11: >>> 0000000000000293 R12: 0000000000000000 >>> Sep 19 22:01:51 maverick kernel: R13: 000073096826cd98 R14: >>> 0000000065000070 R15: 0000000000000001 >>> Sep 19 22:01:51 maverick kernel: Code: 00 c3 48 c7 c7 60 90 67 b4 c6 >>> 05 42 ca d8 00 01 e8 dc e5 d7 ff 0f 0b c3 48 c7 c7 90 90 67 b4 c6 05 >>> 2b ca d8 00 01 e8 c6 e5 d7 ff <0f> 0b e9 86 fe ff ff 48 c7 c7 b8 90 >>> 67 b4 c6 05 10 ca d8 00 01 >>> Sep 19 22:01:51 maverick kernel: ---[ end trace dc2e33bbb9167d29 ]--- >> >> Suricata works correctly and detects attacks as expected. Starting >> sequence took about 83 seconds on my testing hardware. >> >> In my point of view, the ISO includes all necessary fixes for curing >> our Suricata problems - slow establish of new connections, DNS >> trouble, >> et cetera - and I would like them to be included in the next Core >> Update >> (testing) so we can give it a bigger and more extensive test. What >> do you think about this? >> >> OpenVPN performance is poor, but I guess that is because of something >> else. >> >> Command outputs for reference: >>> [root(a)maverick ~]# uname -a >>> Linux maverick 4.14.138-ipfire #1 SMP Mon Sep 9 07:55:34 GMT 2019 >>> x86_64 Intel(R) Celeron(R) CPU N3150 @ 1.60GHz GenuineIntel GNU/Linux >> >>> [root(a)maverick ~]# suricata -V >>> This is Suricata version 4.1.4 RELEASE >> >>> [root(a)maverick ~]# suricata --build-info >>> This is Suricata version 4.1.4 RELEASE >>> Features: NFQ PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG >>> LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_LIBJANSSON TLS >>> MAGIC RUST >>> SIMD support: none >>> Atomic intrisics: 1 2 4 8 byte(s) >>> 64-bits, Little-endian architecture >>> GCC version 8.3.0, C version 199901 >>> compiled with _FORTIFY_SOURCE=2 >>> L1 cache line size (CLS)=64 >>> thread local storage method: __thread >>> compiled with LibHTP v0.5.30, linked against LibHTP v0.5.30 >>> >>> Suricata Configuration: >>> AF_PACKET support: yes >>> eBPF support: no >>> XDP support: no >>> PF_RING support: no >>> NFQueue support: yes >>> NFLOG support: no >>> IPFW support: no >>> Netmap support: no >>> DAG enabled: no >>> Napatech enabled: no >>> WinDivert enabled: no >>> >>> Unix socket enabled: yes >>> Detection enabled: yes >>> >>> Libmagic support: yes >>> libnss support: no >>> libnspr support: no >>> libjansson support: yes >>> liblzma support: yes >>> hiredis support: no >>> hiredis async with libevent: no >>> Prelude support: no >>> PCRE jit: yes >>> LUA support: no >>> libluajit: no >>> libgeoip: no >>> Non-bundled htp: yes >>> Old barnyard2 support: no >>> Hyperscan support: yes >>> Libnet support: yes >>> liblz4 support: no >>> >>> Rust support: yes >>> Rust strict mode: no >>> Rust debug mode: no >>> Rust compiler: rustc 1.37.0 (eae3437df >>> 2019-08-13) >>> Rust cargo: cargo 1.37.0 (9edd08916 >>> 2019-08-02) >>> >>> Install suricatasc: no >>> Install suricata-update: no >>> >>> Profiling enabled: no >>> Profiling locks enabled: no >>> >>> Development settings: >>> Coccinelle / spatch: no >>> Unit tests enabled: no >>> Debug output enabled: no >>> Debug validation enabled: no >>> >>> Generic build parameters: >>> Installation prefix: /usr >>> Configuration directory: /etc/suricata/ >>> Log directory: /var/log/suricata/ >>> >>> --prefix /usr >>> --sysconfdir /etc >>> --localstatedir /var >>> --datarootdir /usr/share >>> >>> Host: x86_64-pc-linux-gnu >>> Compiler: gcc (exec name) / gcc >>> (real) >>> GCC Protect enabled: yes >>> GCC march native enabled: no >>> GCC Profile enabled: no >>> Position Independent Executable enabled: no >>> CFLAGS -O2 -pipe -Wall >>> -fexceptions -fPIC -m64 -mindirect-branch=thunk >>> -mfunction-return=thunk -mtune=generic -Wp,-D_FORTIFY_SOURCE=2 >>> -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong >>> -I${srcdir}/../rust/gen/c-headers >>> PCAP_CFLAGS -I/usr/include >>> SECCFLAGS -fstack-protector >>> -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security >> >> Thanks, and best regards, >> Peter Müller