From: Arne Fitzenreiter <arne_f@ipfire.org>
To: development@lists.ipfire.org
Subject: Re: Testing report regarding image: next-suricata-rust/d22217e1-dirty
Date: Tue, 24 Sep 2019 09:24:25 +0200 [thread overview]
Message-ID: <afa9aa342fbf1f95479ab55e777c6118@ipfire.org> (raw)
In-Reply-To: <81CE3C2B-203D-4E12-8241-4B9C363B57D4@ipfire.org>
[-- Attachment #1: Type: text/plain, Size: 17683 bytes --]
Im not sure but I have found the same trace on my sytem on 1. September
(without the patch)
so it should not caused by the patch itself.
I have also put packages to install patched kernel to my people for
better testing. (i586 currently missing, helena still build it...)
https://people.ipfire.org/~arne_f/highly-experimental/kernel/
Arne
Am 2019-09-24 00:17, schrieb Michael Tremer:
> What are we doing about these kernel traces?
>
> They don’t look like we can ignore them.
>
> -Michael
>
>> On 19 Sep 2019, at 21:16, peter.mueller(a)ipfire.org wrote:
>>
>> Hello Stefan,
>>
>> sorry for my late reply on this topic. Finally, I was able to
>> test your new ISO, next-suricata-rust/d22217e1-dirty .
>>
>> New connections, regardless of SSH, TLS, DNS or whatever, are
>> now established instantly. No packets are dropped anymore, this
>> issue seems to be solved by the changes included in your ISO.
>>
>> OpenVPN RW performance, however, is worse than before: Now around
>> ~ 400 kB/sec. if Suricata is enabled, and somewhere around 900 kB/sec.
>> if it is not. I am getting the feeling this is an OpenVPN bug or
>> performance issue, but as mentioned several times before, I am not
>> satisfied with OpenVPN anyway.
>>
>> A few minutes after booting, kernel emits these log lines:
>>> Sep 19 22:01:51 maverick kernel: refcount_t: increment on 0;
>>> use-after-free.
>>> Sep 19 22:01:51 maverick kernel: ------------[ cut here ]------------
>>> Sep 19 22:01:51 maverick kernel: WARNING: CPU: 2 PID: 2510 at
>>> lib/refcount.c:153 refcount_inc.cold.12+0x13/0x16
>>> Sep 19 22:01:51 maverick kernel: Modules linked in: xt_IMQ imq
>>> xt_length xt_DSCP xt_layer7 cls_fw sch_htb chacha20_x86_64
>>> chacha20_generic poly1305_x86_64 poly1305_generic chacha20poly1305
>>> esp4 xfrm6_mode_tunnel xfrm4_mode_tunnel tun nfnetlink_queue
>>> xt_NFQUEUE ipt_MASQUERADE nf_nat_masquerade_ipv4 pppoe pppox
>>> ppp_generic slhc xt_geoip(O) xt_connlimit xt_multiport xt_hashlimit
>>> xt_mark xt_policy xt_TCPMSS nf_nat_irc nf_conntrack_irc nf_nat_tftp
>>> nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp xt_CT xt_helper
>>> nf_nat_h323 nf_conntrack_h323 xt_conntrack xt_comment ipt_REJECT
>>> nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG xt_limit iptable_raw
>>> iptable_mangle iptable_filter vfat fat snd_hda_codec_hdmi
>>> sch_fq_codel snd_hda_codec_realtek snd_hda_codec_generic
>>> intel_powerclamp coretemp kvm_intel i2c_algo_bit fb_sys_fops
>>> syscopyarea
>>> Sep 19 22:01:51 maverick kernel: sysfillrect sysimgblt snd_hda_intel
>>> kvm snd_hda_codec snd_hda_core iTCO_wdt iTCO_vendor_support irqbypass
>>> snd_hwdep crct10dif_pclmul mcs7830 crc32_pclmul snd_pcm
>>> ghash_clmulni_intel snd_timer usbnet r8169 pcspkr snd i2c_i801
>>> lpc_ich mfd_core mii soundcore i2c_hid i2c_core pcc_cpufreq
>>> rfkill_gpio rfkill intel_int0002_vgpio lp parport_pc parport video
>>> Sep 19 22:01:51 maverick kernel: CPU: 2 PID: 2510 Comm: W-Q3 Tainted:
>>> G O 4.14.138-ipfire #1
>>> Sep 19 22:01:51 maverick kernel: Hardware name: Gigabyte Technology
>>> Co., Ltd. Default string/N3150ND3V, BIOS F5a 01/19/2018
>>> Sep 19 22:01:51 maverick kernel: task: ffffa393374fb200 task.stack:
>>> ffffa98f00290000
>>> Sep 19 22:01:51 maverick kernel: RIP:
>>> 0010:refcount_inc.cold.12+0x13/0x16
>>> Sep 19 22:01:51 maverick kernel: RSP: 0018:ffffa98f00293798 EFLAGS:
>>> 00010246
>>> Sep 19 22:01:51 maverick kernel: RAX: 000000000000002b RBX:
>>> ffffa39338e7ed00 RCX: 0000000000000000
>>> Sep 19 22:01:51 maverick kernel: RDX: 0000000000000000 RSI:
>>> ffffa3933fd163f8 RDI: ffffa3933fd163f8
>>> Sep 19 22:01:51 maverick kernel: RBP: ffffffffb3e9e220 R08:
>>> 000000000000003c R09: 000000000000027c
>>> Sep 19 22:01:51 maverick kernel: R10: 0000000000000000 R11:
>>> 0000000000000001 R12: ffffa39338c81100
>>> Sep 19 22:01:51 maverick kernel: R13: ffffffffb44c35c0 R14:
>>> 0000000000028003 R15: ffffffffc03fc3e0
>>> Sep 19 22:01:51 maverick kernel: FS: 00007309737fe700(0000)
>>> GS:ffffa3933fd00000(0000) knlGS:0000000000000000
>>> Sep 19 22:01:51 maverick kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
>>> 0000000080050033
>>> Sep 19 22:01:51 maverick kernel: CR2: 00007eb62b581728 CR3:
>>> 0000000177e30000 CR4: 00000000001006e0
>>> Sep 19 22:01:51 maverick kernel: Call Trace:
>>> Sep 19 22:01:51 maverick kernel: nf_queue_entry_get_refs+0x41/0x90
>>> Sep 19 22:01:51 maverick kernel: nf_queue+0xf9/0x220
>>> Sep 19 22:01:51 maverick kernel: nf_hook_slow+0x9f/0xf0
>>> Sep 19 22:01:51 maverick kernel: __ip_local_out+0xe4/0x150
>>> Sep 19 22:01:51 maverick kernel: ?
>>> ip_forward_options.cold.7+0x27/0x27
>>> Sep 19 22:01:51 maverick kernel: xfrm_output_resume+0x21e/0x540
>>> Sep 19 22:01:51 maverick kernel: ? ipv4_confirm+0x3f/0xd0
>>> Sep 19 22:01:51 maverick kernel: xfrm4_output+0x3a/0xe0
>>> Sep 19 22:01:51 maverick kernel: ? xfrm4_udp_encap_rcv+0x1a0/0x1a0
>>> Sep 19 22:01:51 maverick kernel: nf_reinject+0x176/0x190
>>> Sep 19 22:01:51 maverick kernel: nfqnl_recv_verdict+0x293/0x4a0
>>> [nfnetlink_queue]
>>> Sep 19 22:01:51 maverick kernel: ? nla_parse+0xb5/0xe0
>>> Sep 19 22:01:51 maverick kernel: nfnetlink_rcv_msg+0x14e/0x260
>>> Sep 19 22:01:51 maverick kernel: ?
>>> nfnetlink_net_exit_batch+0x60/0x60
>>> Sep 19 22:01:51 maverick kernel: netlink_rcv_skb+0x78/0x150
>>> Sep 19 22:01:51 maverick kernel: nfnetlink_rcv+0x70/0x760
>>> Sep 19 22:01:51 maverick kernel: ? lock_timer_base+0x67/0x80
>>> Sep 19 22:01:51 maverick kernel: ? try_to_del_timer_sync+0x4d/0x80
>>> Sep 19 22:01:51 maverick kernel: ? __netlink_lookup+0xe1/0x140
>>> Sep 19 22:01:51 maverick kernel: netlink_unicast+0x183/0x230
>>> Sep 19 22:01:51 maverick kernel: netlink_sendmsg+0x204/0x3d0
>>> Sep 19 22:01:51 maverick kernel: sock_sendmsg+0x36/0x40
>>> Sep 19 22:01:51 maverick kernel: ___sys_sendmsg+0x2a7/0x300
>>> Sep 19 22:01:51 maverick kernel: ? netlink_recvmsg+0x398/0x460
>>> Sep 19 22:01:51 maverick kernel: ? __switch_to_asm+0x41/0x70
>>> Sep 19 22:01:51 maverick kernel: __sys_sendmsg+0x67/0xb0
>>> Sep 19 22:01:51 maverick kernel: do_syscall_64+0x67/0x100
>>> Sep 19 22:01:51 maverick kernel:
>>> entry_SYSCALL_64_after_hwframe+0x3d/0xa2
>>> Sep 19 22:01:51 maverick kernel: RIP: 0033:0x73097af4a8c7
>>> Sep 19 22:01:51 maverick kernel: RSP: 002b:00007309737fbfc0 EFLAGS:
>>> 00000293 ORIG_RAX: 000000000000002e
>>> Sep 19 22:01:51 maverick kernel: RAX: ffffffffffffffda RBX:
>>> 0000000000000008 RCX: 000073097af4a8c7
>>> Sep 19 22:01:51 maverick kernel: RDX: 0000000000000000 RSI:
>>> 00007309737fc000 RDI: 0000000000000008
>>> Sep 19 22:01:51 maverick kernel: RBP: 00007309737fc000 R08:
>>> 0000000000000000 R09: 0000000000000301
>>> Sep 19 22:01:51 maverick kernel: R10: 0000730974409e20 R11:
>>> 0000000000000293 R12: 0000000000000000
>>> Sep 19 22:01:51 maverick kernel: R13: 000073096026cd98 R14:
>>> 0000000000000070 R15: 0000000000000001
>>> Sep 19 22:01:51 maverick kernel: Code: 08 90 67 b4 c6 05 5e ca d8 00
>>> 01 e8 f7 e5 d7 ff 0f 0b b8 01 00 00 00 c3 48 c7 c7 60 90 67 b4 c6 05
>>> 42 ca d8 00 01 e8 dc e5 d7 ff <0f> 0b c3 48 c7 c7 90 90 67 b4 c6 05
>>> 2b ca d8 00 01 e8 c6 e5 d7
>>> Sep 19 22:01:51 maverick kernel: ---[ end trace dc2e33bbb9167d28 ]---
>>> Sep 19 22:01:51 maverick kernel: refcount_t: underflow;
>>> use-after-free.
>>> Sep 19 22:01:51 maverick kernel: ------------[ cut here ]------------
>>> Sep 19 22:01:51 maverick kernel: WARNING: CPU: 3 PID: 2507 at
>>> lib/refcount.c:187 refcount_sub_and_test.cold.13+0x13/0x1a
>>> Sep 19 22:01:51 maverick kernel: Modules linked in: xt_IMQ imq
>>> xt_length xt_DSCP xt_layer7 cls_fw sch_htb chacha20_x86_64
>>> chacha20_generic poly1305_x86_64 poly1305_generic chacha20poly1305
>>> esp4 xfrm6_mode_tunnel xfrm4_mode_tunnel tun nfnetlink_queue
>>> xt_NFQUEUE ipt_MASQUERADE nf_nat_masquerade_ipv4 pppoe pppox
>>> ppp_generic slhc xt_geoip(O) xt_connlimit xt_multiport xt_hashlimit
>>> xt_mark xt_policy xt_TCPMSS nf_nat_irc nf_conntrack_irc nf_nat_tftp
>>> nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp xt_CT xt_helper
>>> nf_nat_h323 nf_conntrack_h323 xt_conntrack xt_comment ipt_REJECT
>>> nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG xt_limit iptable_raw
>>> iptable_mangle iptable_filter vfat fat snd_hda_codec_hdmi
>>> sch_fq_codel snd_hda_codec_realtek snd_hda_codec_generic
>>> intel_powerclamp coretemp kvm_intel i2c_algo_bit fb_sys_fops
>>> syscopyarea
>>> Sep 19 22:01:51 maverick kernel: sysfillrect sysimgblt snd_hda_intel
>>> kvm snd_hda_codec snd_hda_core iTCO_wdt iTCO_vendor_support irqbypass
>>> snd_hwdep crct10dif_pclmul mcs7830 crc32_pclmul snd_pcm
>>> ghash_clmulni_intel snd_timer usbnet r8169 pcspkr snd i2c_i801
>>> lpc_ich mfd_core mii soundcore i2c_hid i2c_core pcc_cpufreq
>>> rfkill_gpio rfkill intel_int0002_vgpio lp parport_pc parport video
>>> Sep 19 22:01:51 maverick kernel: CPU: 3 PID: 2507 Comm: W-Q2 Tainted:
>>> G W O 4.14.138-ipfire #1
>>> Sep 19 22:01:51 maverick kernel: Hardware name: Gigabyte Technology
>>> Co., Ltd. Default string/N3150ND3V, BIOS F5a 01/19/2018
>>> Sep 19 22:01:51 maverick kernel: task: ffffa393395ea580 task.stack:
>>> ffffa98f00250000
>>> Sep 19 22:01:51 maverick kernel: RIP:
>>> 0010:refcount_sub_and_test.cold.13+0x13/0x1a
>>> Sep 19 22:01:51 maverick kernel: RSP: 0018:ffffa98f00253928 EFLAGS:
>>> 00010246
>>> Sep 19 22:01:51 maverick kernel: RAX: 0000000000000026 RBX:
>>> 0000000000000000 RCX: 0000000000000000
>>> Sep 19 22:01:51 maverick kernel: RDX: 0000000000000000 RSI:
>>> ffffa3933fd963f8 RDI: ffffa3933fd963f8
>>> Sep 19 22:01:51 maverick kernel: RBP: ffffa39339bc4400 R08:
>>> 0000000000000038 R09: 00000000000002b4
>>> Sep 19 22:01:51 maverick kernel: R10: 0000000000000000 R11:
>>> 0000000000000001 R12: ffffa393383ab480
>>> Sep 19 22:01:51 maverick kernel: R13: ffffa39337f5da00 R14:
>>> ffffa39339cd9840 R15: 0000000000000000
>>> Sep 19 22:01:51 maverick kernel: FS: 0000730973fff700(0000)
>>> GS:ffffa3933fd80000(0000) knlGS:0000000000000000
>>> Sep 19 22:01:51 maverick kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
>>> 0000000080050033
>>> Sep 19 22:01:51 maverick kernel: CR2: 0000000002218828 CR3:
>>> 0000000177e30000 CR4: 00000000001006e0
>>> Sep 19 22:01:51 maverick kernel: Call Trace:
>>> Sep 19 22:01:51 maverick kernel:
>>> nf_queue_entry_release_refs+0x45/0xa0
>>> Sep 19 22:01:51 maverick kernel: nf_reinject+0x3d/0x190
>>> Sep 19 22:01:51 maverick kernel: nfqnl_recv_verdict+0x293/0x4a0
>>> [nfnetlink_queue]
>>> Sep 19 22:01:51 maverick kernel: ? nla_parse+0xb5/0xe0
>>> Sep 19 22:01:51 maverick kernel: nfnetlink_rcv_msg+0x14e/0x260
>>> Sep 19 22:01:51 maverick kernel: ?
>>> nfnetlink_net_exit_batch+0x60/0x60
>>> Sep 19 22:01:51 maverick kernel: netlink_rcv_skb+0x78/0x150
>>> Sep 19 22:01:51 maverick kernel: nfnetlink_rcv+0x70/0x760
>>> Sep 19 22:01:51 maverick kernel: ? lock_timer_base+0x67/0x80
>>> Sep 19 22:01:51 maverick kernel: ? try_to_del_timer_sync+0x4d/0x80
>>> Sep 19 22:01:51 maverick kernel: ? __netlink_lookup+0xe1/0x140
>>> Sep 19 22:01:51 maverick kernel: netlink_unicast+0x183/0x230
>>> Sep 19 22:01:51 maverick kernel: netlink_sendmsg+0x204/0x3d0
>>> Sep 19 22:01:51 maverick kernel: sock_sendmsg+0x36/0x40
>>> Sep 19 22:01:51 maverick kernel: ___sys_sendmsg+0x2a7/0x300
>>> Sep 19 22:01:51 maverick kernel: ? netlink_recvmsg+0x398/0x460
>>> Sep 19 22:01:51 maverick kernel: ? __switch_to_asm+0x41/0x70
>>> Sep 19 22:01:51 maverick kernel: __sys_sendmsg+0x67/0xb0
>>> Sep 19 22:01:51 maverick kernel: do_syscall_64+0x67/0x100
>>> Sep 19 22:01:51 maverick kernel:
>>> entry_SYSCALL_64_after_hwframe+0x3d/0xa2
>>> Sep 19 22:01:51 maverick kernel: RIP: 0033:0x73097af4a8c7
>>> Sep 19 22:01:51 maverick kernel: RSP: 002b:0000730973ffcfc0 EFLAGS:
>>> 00000293 ORIG_RAX: 000000000000002e
>>> Sep 19 22:01:51 maverick kernel: RAX: ffffffffffffffda RBX:
>>> 0000000000000007 RCX: 000073097af4a8c7
>>> Sep 19 22:01:51 maverick kernel: RDX: 0000000000000000 RSI:
>>> 0000730973ffd000 RDI: 0000000000000007
>>> Sep 19 22:01:51 maverick kernel: RBP: 0000730973ffd000 R08:
>>> 0000000000000000 R09: 0000000000000301
>>> Sep 19 22:01:51 maverick kernel: R10: 000073096826c188 R11:
>>> 0000000000000293 R12: 0000000000000000
>>> Sep 19 22:01:51 maverick kernel: R13: 000073096826cd98 R14:
>>> 0000000065000070 R15: 0000000000000001
>>> Sep 19 22:01:51 maverick kernel: Code: 00 c3 48 c7 c7 60 90 67 b4 c6
>>> 05 42 ca d8 00 01 e8 dc e5 d7 ff 0f 0b c3 48 c7 c7 90 90 67 b4 c6 05
>>> 2b ca d8 00 01 e8 c6 e5 d7 ff <0f> 0b e9 86 fe ff ff 48 c7 c7 b8 90
>>> 67 b4 c6 05 10 ca d8 00 01
>>> Sep 19 22:01:51 maverick kernel: ---[ end trace dc2e33bbb9167d29 ]---
>>
>> Suricata works correctly and detects attacks as expected. Starting
>> sequence took about 83 seconds on my testing hardware.
>>
>> In my point of view, the ISO includes all necessary fixes for curing
>> our Suricata problems - slow establish of new connections, DNS
>> trouble,
>> et cetera - and I would like them to be included in the next Core
>> Update
>> (testing) so we can give it a bigger and more extensive test. What
>> do you think about this?
>>
>> OpenVPN performance is poor, but I guess that is because of something
>> else.
>>
>> Command outputs for reference:
>>> [root(a)maverick ~]# uname -a
>>> Linux maverick 4.14.138-ipfire #1 SMP Mon Sep 9 07:55:34 GMT 2019
>>> x86_64 Intel(R) Celeron(R) CPU N3150 @ 1.60GHz GenuineIntel GNU/Linux
>>
>>> [root(a)maverick ~]# suricata -V
>>> This is Suricata version 4.1.4 RELEASE
>>
>>> [root(a)maverick ~]# suricata --build-info
>>> This is Suricata version 4.1.4 RELEASE
>>> Features: NFQ PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG
>>> LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_LIBJANSSON TLS
>>> MAGIC RUST
>>> SIMD support: none
>>> Atomic intrisics: 1 2 4 8 byte(s)
>>> 64-bits, Little-endian architecture
>>> GCC version 8.3.0, C version 199901
>>> compiled with _FORTIFY_SOURCE=2
>>> L1 cache line size (CLS)=64
>>> thread local storage method: __thread
>>> compiled with LibHTP v0.5.30, linked against LibHTP v0.5.30
>>>
>>> Suricata Configuration:
>>> AF_PACKET support: yes
>>> eBPF support: no
>>> XDP support: no
>>> PF_RING support: no
>>> NFQueue support: yes
>>> NFLOG support: no
>>> IPFW support: no
>>> Netmap support: no
>>> DAG enabled: no
>>> Napatech enabled: no
>>> WinDivert enabled: no
>>>
>>> Unix socket enabled: yes
>>> Detection enabled: yes
>>>
>>> Libmagic support: yes
>>> libnss support: no
>>> libnspr support: no
>>> libjansson support: yes
>>> liblzma support: yes
>>> hiredis support: no
>>> hiredis async with libevent: no
>>> Prelude support: no
>>> PCRE jit: yes
>>> LUA support: no
>>> libluajit: no
>>> libgeoip: no
>>> Non-bundled htp: yes
>>> Old barnyard2 support: no
>>> Hyperscan support: yes
>>> Libnet support: yes
>>> liblz4 support: no
>>>
>>> Rust support: yes
>>> Rust strict mode: no
>>> Rust debug mode: no
>>> Rust compiler: rustc 1.37.0 (eae3437df
>>> 2019-08-13)
>>> Rust cargo: cargo 1.37.0 (9edd08916
>>> 2019-08-02)
>>>
>>> Install suricatasc: no
>>> Install suricata-update: no
>>>
>>> Profiling enabled: no
>>> Profiling locks enabled: no
>>>
>>> Development settings:
>>> Coccinelle / spatch: no
>>> Unit tests enabled: no
>>> Debug output enabled: no
>>> Debug validation enabled: no
>>>
>>> Generic build parameters:
>>> Installation prefix: /usr
>>> Configuration directory: /etc/suricata/
>>> Log directory: /var/log/suricata/
>>>
>>> --prefix /usr
>>> --sysconfdir /etc
>>> --localstatedir /var
>>> --datarootdir /usr/share
>>>
>>> Host: x86_64-pc-linux-gnu
>>> Compiler: gcc (exec name) / gcc
>>> (real)
>>> GCC Protect enabled: yes
>>> GCC march native enabled: no
>>> GCC Profile enabled: no
>>> Position Independent Executable enabled: no
>>> CFLAGS -O2 -pipe -Wall
>>> -fexceptions -fPIC -m64 -mindirect-branch=thunk
>>> -mfunction-return=thunk -mtune=generic -Wp,-D_FORTIFY_SOURCE=2
>>> -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong
>>> -I${srcdir}/../rust/gen/c-headers
>>> PCAP_CFLAGS -I/usr/include
>>> SECCFLAGS -fstack-protector
>>> -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security
>>
>> Thanks, and best regards,
>> Peter Müller
next prev parent reply other threads:[~2019-09-24 7:24 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-09-19 20:16 peter.mueller
2019-09-23 22:17 ` Michael Tremer
2019-09-24 7:24 ` Arne Fitzenreiter [this message]
2019-09-24 10:15 ` Michael Tremer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=afa9aa342fbf1f95479ab55e777c6118@ipfire.org \
--to=arne_f@ipfire.org \
--cc=development@lists.ipfire.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox