From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: Testing report regarding image: next-suricata-rust/d22217e1-dirty Date: Tue, 24 Sep 2019 11:15:31 +0100 Message-ID: <45C4FD2A-DCB2-4F94-9814-231AF38847ED@ipfire.org> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============3965422997464012877==" List-Id: --===============3965422997464012877== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi, > On 24 Sep 2019, at 08:24, Arne Fitzenreiter wrote: >=20 > Im not sure but I have found the same trace on my sytem on 1. September (wi= thout the patch) > so it should not caused by the patch itself. I did not assume that it comes from this patch, but they definitely do not lo= ok good. Do we wait until they are being fixed upstream or can we solve these ourselve= s easily? Best, -Michael > I have also put packages to install patched kernel to my people for > better testing. (i586 currently missing, helena still build it...) >=20 > https://people.ipfire.org/~arne_f/highly-experimental/kernel/ >=20 > Arne >=20 > Am 2019-09-24 00:17, schrieb Michael Tremer: >> What are we doing about these kernel traces? >> They don=E2=80=99t look like we can ignore them. >> -Michael >>> On 19 Sep 2019, at 21:16, peter.mueller(a)ipfire.org wrote: >>> Hello Stefan, >>> sorry for my late reply on this topic. Finally, I was able to >>> test your new ISO, next-suricata-rust/d22217e1-dirty . >>> New connections, regardless of SSH, TLS, DNS or whatever, are >>> now established instantly. No packets are dropped anymore, this >>> issue seems to be solved by the changes included in your ISO. >>> OpenVPN RW performance, however, is worse than before: Now around >>> ~ 400 kB/sec. if Suricata is enabled, and somewhere around 900 kB/sec. >>> if it is not. I am getting the feeling this is an OpenVPN bug or >>> performance issue, but as mentioned several times before, I am not >>> satisfied with OpenVPN anyway. >>> A few minutes after booting, kernel emits these log lines: >>>> Sep 19 22:01:51 maverick kernel: refcount_t: increment on 0; use-after-f= ree. >>>> Sep 19 22:01:51 maverick kernel: ------------[ cut here ]------------ >>>> Sep 19 22:01:51 maverick kernel: WARNING: CPU: 2 PID: 2510 at lib/refcou= nt.c:153 refcount_inc.cold.12+0x13/0x16 >>>> Sep 19 22:01:51 maverick kernel: Modules linked in: xt_IMQ imq xt_length= xt_DSCP xt_layer7 cls_fw sch_htb chacha20_x86_64 chacha20_generic poly1305_x= 86_64 poly1305_generic chacha20poly1305 esp4 xfrm6_mode_tunnel xfrm4_mode_tun= nel tun nfnetlink_queue xt_NFQUEUE ipt_MASQUERADE nf_nat_masquerade_ipv4 pppo= e pppox ppp_generic slhc xt_geoip(O) xt_connlimit xt_multiport xt_hashlimit x= t_mark xt_policy xt_TCPMSS nf_nat_irc nf_conntrack_irc nf_nat_tftp nf_conntra= ck_tftp nf_nat_ftp nf_conntrack_ftp xt_CT xt_helper nf_nat_h323 nf_conntrack_= h323 xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_com= mon xt_LOG xt_limit iptable_raw iptable_mangle iptable_filter vfat fat snd_hd= a_codec_hdmi sch_fq_codel snd_hda_codec_realtek snd_hda_codec_generic intel_p= owerclamp coretemp kvm_intel i2c_algo_bit fb_sys_fops syscopyarea >>>> Sep 19 22:01:51 maverick kernel: sysfillrect sysimgblt snd_hda_intel kv= m snd_hda_codec snd_hda_core iTCO_wdt iTCO_vendor_support irqbypass snd_hwdep= crct10dif_pclmul mcs7830 crc32_pclmul snd_pcm ghash_clmulni_intel snd_timer = usbnet r8169 pcspkr snd i2c_i801 lpc_ich mfd_core mii soundcore i2c_hid i2c_c= ore pcc_cpufreq rfkill_gpio rfkill intel_int0002_vgpio lp parport_pc parport = video >>>> Sep 19 22:01:51 maverick kernel: CPU: 2 PID: 2510 Comm: W-Q3 Tainted: G = O 4.14.138-ipfire #1 >>>> Sep 19 22:01:51 maverick kernel: Hardware name: Gigabyte Technology Co.,= Ltd. Default string/N3150ND3V, BIOS F5a 01/19/2018 >>>> Sep 19 22:01:51 maverick kernel: task: ffffa393374fb200 task.stack: ffff= a98f00290000 >>>> Sep 19 22:01:51 maverick kernel: RIP: 0010:refcount_inc.cold.12+0x13/0x16 >>>> Sep 19 22:01:51 maverick kernel: RSP: 0018:ffffa98f00293798 EFLAGS: 0001= 0246 >>>> Sep 19 22:01:51 maverick kernel: RAX: 000000000000002b RBX: ffffa39338e7= ed00 RCX: 0000000000000000 >>>> Sep 19 22:01:51 maverick kernel: RDX: 0000000000000000 RSI: ffffa3933fd1= 63f8 RDI: ffffa3933fd163f8 >>>> Sep 19 22:01:51 maverick kernel: RBP: ffffffffb3e9e220 R08: 000000000000= 003c R09: 000000000000027c >>>> Sep 19 22:01:51 maverick kernel: R10: 0000000000000000 R11: 000000000000= 0001 R12: ffffa39338c81100 >>>> Sep 19 22:01:51 maverick kernel: R13: ffffffffb44c35c0 R14: 000000000002= 8003 R15: ffffffffc03fc3e0 >>>> Sep 19 22:01:51 maverick kernel: FS: 00007309737fe700(0000) GS:ffffa393= 3fd00000(0000) knlGS:0000000000000000 >>>> Sep 19 22:01:51 maverick kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000= 0080050033 >>>> Sep 19 22:01:51 maverick kernel: CR2: 00007eb62b581728 CR3: 0000000177e3= 0000 CR4: 00000000001006e0 >>>> Sep 19 22:01:51 maverick kernel: Call Trace: >>>> Sep 19 22:01:51 maverick kernel: nf_queue_entry_get_refs+0x41/0x90 >>>> Sep 19 22:01:51 maverick kernel: nf_queue+0xf9/0x220 >>>> Sep 19 22:01:51 maverick kernel: nf_hook_slow+0x9f/0xf0 >>>> Sep 19 22:01:51 maverick kernel: __ip_local_out+0xe4/0x150 >>>> Sep 19 22:01:51 maverick kernel: ? ip_forward_options.cold.7+0x27/0x27 >>>> Sep 19 22:01:51 maverick kernel: xfrm_output_resume+0x21e/0x540 >>>> Sep 19 22:01:51 maverick kernel: ? ipv4_confirm+0x3f/0xd0 >>>> Sep 19 22:01:51 maverick kernel: xfrm4_output+0x3a/0xe0 >>>> Sep 19 22:01:51 maverick kernel: ? xfrm4_udp_encap_rcv+0x1a0/0x1a0 >>>> Sep 19 22:01:51 maverick kernel: nf_reinject+0x176/0x190 >>>> Sep 19 22:01:51 maverick kernel: nfqnl_recv_verdict+0x293/0x4a0 [nfnetl= ink_queue] >>>> Sep 19 22:01:51 maverick kernel: ? nla_parse+0xb5/0xe0 >>>> Sep 19 22:01:51 maverick kernel: nfnetlink_rcv_msg+0x14e/0x260 >>>> Sep 19 22:01:51 maverick kernel: ? nfnetlink_net_exit_batch+0x60/0x60 >>>> Sep 19 22:01:51 maverick kernel: netlink_rcv_skb+0x78/0x150 >>>> Sep 19 22:01:51 maverick kernel: nfnetlink_rcv+0x70/0x760 >>>> Sep 19 22:01:51 maverick kernel: ? lock_timer_base+0x67/0x80 >>>> Sep 19 22:01:51 maverick kernel: ? try_to_del_timer_sync+0x4d/0x80 >>>> Sep 19 22:01:51 maverick kernel: ? __netlink_lookup+0xe1/0x140 >>>> Sep 19 22:01:51 maverick kernel: netlink_unicast+0x183/0x230 >>>> Sep 19 22:01:51 maverick kernel: netlink_sendmsg+0x204/0x3d0 >>>> Sep 19 22:01:51 maverick kernel: sock_sendmsg+0x36/0x40 >>>> Sep 19 22:01:51 maverick kernel: ___sys_sendmsg+0x2a7/0x300 >>>> Sep 19 22:01:51 maverick kernel: ? netlink_recvmsg+0x398/0x460 >>>> Sep 19 22:01:51 maverick kernel: ? __switch_to_asm+0x41/0x70 >>>> Sep 19 22:01:51 maverick kernel: __sys_sendmsg+0x67/0xb0 >>>> Sep 19 22:01:51 maverick kernel: do_syscall_64+0x67/0x100 >>>> Sep 19 22:01:51 maverick kernel: entry_SYSCALL_64_after_hwframe+0x3d/0x= a2 >>>> Sep 19 22:01:51 maverick kernel: RIP: 0033:0x73097af4a8c7 >>>> Sep 19 22:01:51 maverick kernel: RSP: 002b:00007309737fbfc0 EFLAGS: 0000= 0293 ORIG_RAX: 000000000000002e >>>> Sep 19 22:01:51 maverick kernel: RAX: ffffffffffffffda RBX: 000000000000= 0008 RCX: 000073097af4a8c7 >>>> Sep 19 22:01:51 maverick kernel: RDX: 0000000000000000 RSI: 00007309737f= c000 RDI: 0000000000000008 >>>> Sep 19 22:01:51 maverick kernel: RBP: 00007309737fc000 R08: 000000000000= 0000 R09: 0000000000000301 >>>> Sep 19 22:01:51 maverick kernel: R10: 0000730974409e20 R11: 000000000000= 0293 R12: 0000000000000000 >>>> Sep 19 22:01:51 maverick kernel: R13: 000073096026cd98 R14: 000000000000= 0070 R15: 0000000000000001 >>>> Sep 19 22:01:51 maverick kernel: Code: 08 90 67 b4 c6 05 5e ca d8 00 01 = e8 f7 e5 d7 ff 0f 0b b8 01 00 00 00 c3 48 c7 c7 60 90 67 b4 c6 05 42 ca d8 00= 01 e8 dc e5 d7 ff <0f> 0b c3 48 c7 c7 90 90 67 b4 c6 05 2b ca d8 00 01 e8 c6= e5 d7 >>>> Sep 19 22:01:51 maverick kernel: ---[ end trace dc2e33bbb9167d28 ]--- >>>> Sep 19 22:01:51 maverick kernel: refcount_t: underflow; use-after-free. >>>> Sep 19 22:01:51 maverick kernel: ------------[ cut here ]------------ >>>> Sep 19 22:01:51 maverick kernel: WARNING: CPU: 3 PID: 2507 at lib/refcou= nt.c:187 refcount_sub_and_test.cold.13+0x13/0x1a >>>> Sep 19 22:01:51 maverick kernel: Modules linked in: xt_IMQ imq xt_length= xt_DSCP xt_layer7 cls_fw sch_htb chacha20_x86_64 chacha20_generic poly1305_x= 86_64 poly1305_generic chacha20poly1305 esp4 xfrm6_mode_tunnel xfrm4_mode_tun= nel tun nfnetlink_queue xt_NFQUEUE ipt_MASQUERADE nf_nat_masquerade_ipv4 pppo= e pppox ppp_generic slhc xt_geoip(O) xt_connlimit xt_multiport xt_hashlimit x= t_mark xt_policy xt_TCPMSS nf_nat_irc nf_conntrack_irc nf_nat_tftp nf_conntra= ck_tftp nf_nat_ftp nf_conntrack_ftp xt_CT xt_helper nf_nat_h323 nf_conntrack_= h323 xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_com= mon xt_LOG xt_limit iptable_raw iptable_mangle iptable_filter vfat fat snd_hd= a_codec_hdmi sch_fq_codel snd_hda_codec_realtek snd_hda_codec_generic intel_p= owerclamp coretemp kvm_intel i2c_algo_bit fb_sys_fops syscopyarea >>>> Sep 19 22:01:51 maverick kernel: sysfillrect sysimgblt snd_hda_intel kv= m snd_hda_codec snd_hda_core iTCO_wdt iTCO_vendor_support irqbypass snd_hwdep= crct10dif_pclmul mcs7830 crc32_pclmul snd_pcm ghash_clmulni_intel snd_timer = usbnet r8169 pcspkr snd i2c_i801 lpc_ich mfd_core mii soundcore i2c_hid i2c_c= ore pcc_cpufreq rfkill_gpio rfkill intel_int0002_vgpio lp parport_pc parport = video >>>> Sep 19 22:01:51 maverick kernel: CPU: 3 PID: 2507 Comm: W-Q2 Tainted: G = W O 4.14.138-ipfire #1 >>>> Sep 19 22:01:51 maverick kernel: Hardware name: Gigabyte Technology Co.,= Ltd. Default string/N3150ND3V, BIOS F5a 01/19/2018 >>>> Sep 19 22:01:51 maverick kernel: task: ffffa393395ea580 task.stack: ffff= a98f00250000 >>>> Sep 19 22:01:51 maverick kernel: RIP: 0010:refcount_sub_and_test.cold.13= +0x13/0x1a >>>> Sep 19 22:01:51 maverick kernel: RSP: 0018:ffffa98f00253928 EFLAGS: 0001= 0246 >>>> Sep 19 22:01:51 maverick kernel: RAX: 0000000000000026 RBX: 000000000000= 0000 RCX: 0000000000000000 >>>> Sep 19 22:01:51 maverick kernel: RDX: 0000000000000000 RSI: ffffa3933fd9= 63f8 RDI: ffffa3933fd963f8 >>>> Sep 19 22:01:51 maverick kernel: RBP: ffffa39339bc4400 R08: 000000000000= 0038 R09: 00000000000002b4 >>>> Sep 19 22:01:51 maverick kernel: R10: 0000000000000000 R11: 000000000000= 0001 R12: ffffa393383ab480 >>>> Sep 19 22:01:51 maverick kernel: R13: ffffa39337f5da00 R14: ffffa39339cd= 9840 R15: 0000000000000000 >>>> Sep 19 22:01:51 maverick kernel: FS: 0000730973fff700(0000) GS:ffffa393= 3fd80000(0000) knlGS:0000000000000000 >>>> Sep 19 22:01:51 maverick kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000= 0080050033 >>>> Sep 19 22:01:51 maverick kernel: CR2: 0000000002218828 CR3: 0000000177e3= 0000 CR4: 00000000001006e0 >>>> Sep 19 22:01:51 maverick kernel: Call Trace: >>>> Sep 19 22:01:51 maverick kernel: nf_queue_entry_release_refs+0x45/0xa0 >>>> Sep 19 22:01:51 maverick kernel: nf_reinject+0x3d/0x190 >>>> Sep 19 22:01:51 maverick kernel: nfqnl_recv_verdict+0x293/0x4a0 [nfnetl= ink_queue] >>>> Sep 19 22:01:51 maverick kernel: ? nla_parse+0xb5/0xe0 >>>> Sep 19 22:01:51 maverick kernel: nfnetlink_rcv_msg+0x14e/0x260 >>>> Sep 19 22:01:51 maverick kernel: ? nfnetlink_net_exit_batch+0x60/0x60 >>>> Sep 19 22:01:51 maverick kernel: netlink_rcv_skb+0x78/0x150 >>>> Sep 19 22:01:51 maverick kernel: nfnetlink_rcv+0x70/0x760 >>>> Sep 19 22:01:51 maverick kernel: ? lock_timer_base+0x67/0x80 >>>> Sep 19 22:01:51 maverick kernel: ? try_to_del_timer_sync+0x4d/0x80 >>>> Sep 19 22:01:51 maverick kernel: ? __netlink_lookup+0xe1/0x140 >>>> Sep 19 22:01:51 maverick kernel: netlink_unicast+0x183/0x230 >>>> Sep 19 22:01:51 maverick kernel: netlink_sendmsg+0x204/0x3d0 >>>> Sep 19 22:01:51 maverick kernel: sock_sendmsg+0x36/0x40 >>>> Sep 19 22:01:51 maverick kernel: ___sys_sendmsg+0x2a7/0x300 >>>> Sep 19 22:01:51 maverick kernel: ? netlink_recvmsg+0x398/0x460 >>>> Sep 19 22:01:51 maverick kernel: ? __switch_to_asm+0x41/0x70 >>>> Sep 19 22:01:51 maverick kernel: __sys_sendmsg+0x67/0xb0 >>>> Sep 19 22:01:51 maverick kernel: do_syscall_64+0x67/0x100 >>>> Sep 19 22:01:51 maverick kernel: entry_SYSCALL_64_after_hwframe+0x3d/0x= a2 >>>> Sep 19 22:01:51 maverick kernel: RIP: 0033:0x73097af4a8c7 >>>> Sep 19 22:01:51 maverick kernel: RSP: 002b:0000730973ffcfc0 EFLAGS: 0000= 0293 ORIG_RAX: 000000000000002e >>>> Sep 19 22:01:51 maverick kernel: RAX: ffffffffffffffda RBX: 000000000000= 0007 RCX: 000073097af4a8c7 >>>> Sep 19 22:01:51 maverick kernel: RDX: 0000000000000000 RSI: 0000730973ff= d000 RDI: 0000000000000007 >>>> Sep 19 22:01:51 maverick kernel: RBP: 0000730973ffd000 R08: 000000000000= 0000 R09: 0000000000000301 >>>> Sep 19 22:01:51 maverick kernel: R10: 000073096826c188 R11: 000000000000= 0293 R12: 0000000000000000 >>>> Sep 19 22:01:51 maverick kernel: R13: 000073096826cd98 R14: 000000006500= 0070 R15: 0000000000000001 >>>> Sep 19 22:01:51 maverick kernel: Code: 00 c3 48 c7 c7 60 90 67 b4 c6 05 = 42 ca d8 00 01 e8 dc e5 d7 ff 0f 0b c3 48 c7 c7 90 90 67 b4 c6 05 2b ca d8 00= 01 e8 c6 e5 d7 ff <0f> 0b e9 86 fe ff ff 48 c7 c7 b8 90 67 b4 c6 05 10 ca d8= 00 01 >>>> Sep 19 22:01:51 maverick kernel: ---[ end trace dc2e33bbb9167d29 ]--- >>> Suricata works correctly and detects attacks as expected. Starting >>> sequence took about 83 seconds on my testing hardware. >>> In my point of view, the ISO includes all necessary fixes for curing >>> our Suricata problems - slow establish of new connections, DNS trouble, >>> et cetera - and I would like them to be included in the next Core Update >>> (testing) so we can give it a bigger and more extensive test. What >>> do you think about this? >>> OpenVPN performance is poor, but I guess that is because of something >>> else. >>> Command outputs for reference: >>>> [root(a)maverick ~]# uname -a >>>> Linux maverick 4.14.138-ipfire #1 SMP Mon Sep 9 07:55:34 GMT 2019 x86_64= Intel(R) Celeron(R) CPU N3150 @ 1.60GHz GenuineIntel GNU/Linux >>>> [root(a)maverick ~]# suricata -V >>>> This is Suricata version 4.1.4 RELEASE >>>> [root(a)maverick ~]# suricata --build-info >>>> This is Suricata version 4.1.4 RELEASE >>>> Features: NFQ PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNE= T1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_LIBJANSSON TLS MAGIC RUST >>>> SIMD support: none >>>> Atomic intrisics: 1 2 4 8 byte(s) >>>> 64-bits, Little-endian architecture >>>> GCC version 8.3.0, C version 199901 >>>> compiled with _FORTIFY_SOURCE=3D2 >>>> L1 cache line size (CLS)=3D64 >>>> thread local storage method: __thread >>>> compiled with LibHTP v0.5.30, linked against LibHTP v0.5.30 >>>> Suricata Configuration: >>>> AF_PACKET support: yes >>>> eBPF support: no >>>> XDP support: no >>>> PF_RING support: no >>>> NFQueue support: yes >>>> NFLOG support: no >>>> IPFW support: no >>>> Netmap support: no >>>> DAG enabled: no >>>> Napatech enabled: no >>>> WinDivert enabled: no >>>> Unix socket enabled: yes >>>> Detection enabled: yes >>>> Libmagic support: yes >>>> libnss support: no >>>> libnspr support: no >>>> libjansson support: yes >>>> liblzma support: yes >>>> hiredis support: no >>>> hiredis async with libevent: no >>>> Prelude support: no >>>> PCRE jit: yes >>>> LUA support: no >>>> libluajit: no >>>> libgeoip: no >>>> Non-bundled htp: yes >>>> Old barnyard2 support: no >>>> Hyperscan support: yes >>>> Libnet support: yes >>>> liblz4 support: no >>>> Rust support: yes >>>> Rust strict mode: no >>>> Rust debug mode: no >>>> Rust compiler: rustc 1.37.0 (eae3437df 2019-08= -13) >>>> Rust cargo: cargo 1.37.0 (9edd08916 2019-08= -02) >>>> Install suricatasc: no >>>> Install suricata-update: no >>>> Profiling enabled: no >>>> Profiling locks enabled: no >>>> Development settings: >>>> Coccinelle / spatch: no >>>> Unit tests enabled: no >>>> Debug output enabled: no >>>> Debug validation enabled: no >>>> Generic build parameters: >>>> Installation prefix: /usr >>>> Configuration directory: /etc/suricata/ >>>> Log directory: /var/log/suricata/ >>>> --prefix /usr >>>> --sysconfdir /etc >>>> --localstatedir /var >>>> --datarootdir /usr/share >>>> Host: x86_64-pc-linux-gnu >>>> Compiler: gcc (exec name) / gcc (real) >>>> GCC Protect enabled: yes >>>> GCC march native enabled: no >>>> GCC Profile enabled: no >>>> Position Independent Executable enabled: no >>>> CFLAGS -O2 -pipe -Wall -fexceptions -f= PIC -m64 -mindirect-branch=3Dthunk -mfunction-return=3Dthunk -mtune=3Dgeneric= -Wp,-D_FORTIFY_SOURCE=3D2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong= -I${srcdir}/../rust/gen/c-headers >>>> PCAP_CFLAGS -I/usr/include >>>> SECCFLAGS -fstack-protector -D_FORTIFY_SO= URCE=3D2 -Wformat -Wformat-security >>> Thanks, and best regards, >>> Peter M=C3=BCller --===============3965422997464012877==--