Dear OSIF/Suricata users,
earlier this year, the linux-based open source firewall distribution IPFire has migrated from Snort to Suricata for a number of reasons (further information is available at https://blog.ipfire.org/post/introducing-ipfire-s-new-intrusion-prevention-s...).
While we are quite pleased with some of its features (multi-threading, ability to monitor several interfaces per process, etc.), we experienced some problems ever since we are running it. Not being reproducible everywhere, we initially thought they were corner cases in obscure network scenarios.
Ultimately, they were not. Even worse, no dropped packets were logged although we can tell for sure there were some.
For example, several IPFire users - including myself - report very slow DNS resolution when trying to access a website, while "normal" lookups using dig or host commands perform fine.
Another issue is reduced OpenVPN tunnel throughput, which seems to be caused by massive packet loss when Suricata is enabled (~ 800 kB/s, ~ 2 MB/s if Suricata is turned off). In order to get closer to its origin, we spend a lot of time on testing and debugging, eventually left without any idea what the solution might be.
Both issues - possibly being related to each other - can be reproduced using Suricata 4.1.4 without any rules or packet decoders enabled. Unfortunately, our setup, where Suricata runs inline via Netfilter queue, does not seem to be documented very well.
That's why I am asking here if anybody is able to tell us what we are doing wrong. Perhaps this just might be a configuration problem, but we are out of ideas where to look for it.
Please find our suricata.yaml (decoders enabled, but disabling it does not matter) and the stats.log file enclosed.
Details regarding a testing machine:
[root@maverick ~]# suricata -V This is Suricata version 4.1.4 RELEASE
[root@maverick ~]# uname -a Linux maverick 4.14.138-ipfire #1 SMP Sat Aug 10 00:53:30 GMT 2019 x86_64 Intel(R) Celeron(R) CPU N3150 @ 1.60GHz GenuineIntel GNU/Linux
[root@maverick ~]# ldd /usr/bin/suricata linux-vdso.so.1 (0x00007ffdd77df000) libdl.so.2 => /lib/libdl.so.2 (0x00007ba2d3e7f000) librt.so.1 => /lib/librt.so.1 (0x00007ba2d3e75000) libm.so.6 => /lib/libm.so.6 (0x00007ba2d3d26000) libmagic.so.1 => /usr/lib/libmagic.so.1 (0x00007ba2d3afc000) libcap-ng.so.0 => /usr/lib/libcap-ng.so.0 (0x00007ba2d38f6000) libpcap.so.1 => /usr/lib/libpcap.so.1 (0x00007ba2d36b5000) libnet.so.1 => /usr/lib/libnet.so.1 (0x00007ba2d3498000) libnetfilter_queue.so.1 => /usr/lib/libnetfilter_queue.so.1 (0x00007ba2d3291000) libnfnetlink.so.0 => /usr/lib/libnfnetlink.so.0 (0x00007ba2d308a000) libjansson.so.4 => /usr/lib/libjansson.so.4 (0x00007ba2d307b000) libpthread.so.0 => /lib/libpthread.so.0 (0x00007ba2d305a000) libyaml-0.so.2 => /usr/lib/libyaml-0.so.2 (0x00007ba2d2e3c000) libpcre.so.1 => /usr/lib/libpcre.so.1 (0x00007ba2d2bc7000) liblzma.so.5 => /usr/lib/liblzma.so.5 (0x00007ba2d29a0000) libhs.so.5 => /usr/lib/libhs.so.5 (0x00007ba2d221a000) libhtp.so.2 => /usr/lib/libhtp.so.2 (0x00007ba2d1ff2000) libc.so.6 => /lib/libc.so.6 (0x00007ba2d1e0d000) /lib64/ld-linux-x86-64.so.2 (0x00007ba2d3e8f000) libmnl.so.0 => /usr/lib/libmnl.so.0 (0x00007ba2d1c07000) libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007ba2d1a6c000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007ba2d1855000) [root@maverick ~]# suricata --build-info This is Suricata version 4.1.4 RELEASE Features: NFQ PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_LIBJANSSON TLS MAGIC SIMD support: none Atomic intrisics: 1 2 4 8 byte(s) 64-bits, Little-endian architecture GCC version 8.3.0, C version 199901 compiled with _FORTIFY_SOURCE=2 L1 cache line size (CLS)=64 thread local storage method: __thread compiled with LibHTP v0.5.28, linked against LibHTP v0.5.28
Suricata Configuration: AF_PACKET support: yes eBPF support: no XDP support: no PF_RING support: no NFQueue support: yes NFLOG support: no IPFW support: no Netmap support: no DAG enabled: no Napatech enabled: no WinDivert enabled: no
Unix socket enabled: yes Detection enabled: yes
Libmagic support: yes libnss support: no libnspr support: no libjansson support: yes liblzma support: yes hiredis support: no hiredis async with libevent: no Prelude support: no PCRE jit: yes LUA support: no libluajit: no libgeoip: no Non-bundled htp: yes Old barnyard2 support: no Hyperscan support: yes Libnet support: yes liblz4 support: no
Rust support: no Rust strict mode: no Rust debug mode: no Rust compiler: not set Rust cargo: not set
Install suricatasc: no Install suricata-update: no
Profiling enabled: no Profiling locks enabled: no
Development settings: Coccinelle / spatch: no Unit tests enabled: no Debug output enabled: no Debug validation enabled: no
Generic build parameters: Installation prefix: /usr Configuration directory: /etc/suricata/ Log directory: /var/log/suricata/
--prefix /usr --sysconfdir /etc --localstatedir /var --datarootdir /usr/share
Host: x86_64-pc-linux-gnu Compiler: gcc (exec name) / gcc (real) GCC Protect enabled: yes GCC march native enabled: no GCC Profile enabled: no Position Independent Executable enabled: no CFLAGS -O2 -pipe -Wall -fexceptions -fPIC -m64 -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong PCAP_CFLAGS -I/usr/include SECCFLAGS -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security
Please let me know if further information is needed. Any help is highly appreciated.
Thanks, and best regards, Peter Müller