Hello Stefan, hello Peter, hello *,
@Stefan: Thank you again for building the ISO with Suricata 5.0.0-beta1, Rust and current libhtp.
@Peter: Sorry for having not answered your question: The problem is not only related to DNS traffic, but to new connections in general (no matter if they are encrypted or plain text) - see below for details.
Initially, Suricata refuses to start:
Sep 8 11:59:43 maverick suricata: [ERRCODE: SC_WARN_NO_STATS_LOGGERS(261)] - stats are enabled but no loggers are active Sep 8 11:59:43 maverick suricata: [ERRCODE: SC_ERR_NFQ_OPEN(68)] - no queue for given index Sep 8 11:59:43 maverick suricata: [ERRCODE: SC_ERR_NFQ_THREAD_INIT(78)] - nfq thread failed to initialize
Although statistics are disabled in /etc/suricata/suricata.yaml , enabling the statistics logger was necessary. Perhaps a glitch in the beta version, as the statistics log file is empty.
Instead of passing the NFQ indices one by one ("-q 0 -q 1 -q 2 -q 3"), Suricata now likes them as a range: "-q 0:3" After changing this in the initscript and deleting orphaned PID file, Suricata starts correctly.
Initialisation procedure takes 87 seconds on my testing hardware, which is approximately two times faster compared to Suricata 4.x. Rule parsing works, all tested attacks were successfully detected.
Resource consumption of Suricata 5.x is a bit lower compared to 4.x .
Unfortunately, both of my problems can be reproduced with that image: (a) Poor OpenVPN throughput. This has improved a bit to 1.2 MB/sec peak, but still is lower than the 2.1 MB/sec I observe on another productive machine. (b) Establishing connections and DNS resolutions takes age Regardless of SSH, HTTP, HTTPS, SMTPS or IMAPS, establishing a new connection takes 1-2 seconds due to massive packet loss. Resolving DNS records using "dig" or "host" is fast, but using "wget" or "curl" is slow.
Increasing memory allocations or "max-pending-packets" did not help again. That being said, I think we can be more generous regarding our memory allocations, as most RAM of my testing hardware stayed unallocated. :-)
As Eric Leblond already mentioned on the OISF mailing list, the actual problem seems to be something else (netfilter/iptables/?).
Version commands for reference:
[root@maverick ~]# suricata -V This is Suricata version 5.0.0-beta1 RELEASE
[root@maverick ~]# uname -a Linux maverick 4.14.138-ipfire #1 SMP Sat Sep 7 06:27:36 GMT 2019 x86_64 Intel(R) Celeron(R) CPU N3150 @ 1.60GHz GenuineIntel GNU/Linux
[root@maverick ~]# suricata --build-info This is Suricata version 5.0.0-beta1 RELEASE Features: NFQ PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_LIBJANSSON TLS MAGIC RUST SIMD support: none Atomic intrisics: 1 2 4 8 byte(s) 64-bits, Little-endian architecture GCC version 8.3.0, C version 199901 compiled with _FORTIFY_SOURCE=2 L1 cache line size (CLS)=64 thread local storage method: __thread compiled with LibHTP v0.5.30, linked against LibHTP v0.5.30
Suricata Configuration: AF_PACKET support: yes eBPF support: no XDP support: no PF_RING support: no NFQueue support: yes NFLOG support: no IPFW support: no Netmap support: no DAG enabled: no Napatech enabled: no WinDivert enabled: no
Unix socket enabled: yes Detection enabled: yes
Libmagic support: yes libnss support: yes libnspr support: yes libjansson support: yes liblzma support: yes hiredis support: no hiredis async with libevent: no Prelude support: no PCRE jit: yes LUA support: no libluajit: no libgeoip: no Non-bundled htp: yes Old barnyard2 support: no Hyperscan support: yes Libnet support: yes liblz4 support: no
Rust support: yes Rust strict mode: no Rust debug mode: no Rust compiler: rustc 1.37.0 (eae3437df 2019-08-13) Rust cargo: cargo 1.37.0 (9edd08916 2019-08-02)
Python support: no Python path: not set Python version: not set Python distutils no Python yaml no Install suricatactl: requires python Install suricatasc: requires python Install suricata-update: not bundled
Profiling enabled: no Profiling locks enabled: no
Development settings: Coccinelle / spatch: no Unit tests enabled: no Debug output enabled: no Debug validation enabled: no
Generic build parameters: Installation prefix: /usr Configuration directory: /etc/suricata/ Log directory: /var/log/suricata/
--prefix /usr --sysconfdir /etc --localstatedir /var --datarootdir /usr/share
Host: x86_64-pc-linux-gnu Compiler: gcc (exec name) / gcc (real) GCC Protect enabled: yes GCC march native enabled: no GCC Profile enabled: no Position Independent Executable enabled: no CFLAGS -O2 -pipe -Wall -fexceptions -fPIC -m64 -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -I${srcdir}/../rust/gen/c-headers PCAP_CFLAGS -I/usr/include SECCFLAGS -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security
Thanks, and best regards, Peter Müller
Hello Peter, Hello Victor, hello *,
@Victor, thanks for pointing to the bug at your bugtracker.
https://redmine.openinfosecfoundation.org/issues/2806#note-19
I think we are affected exactly by this issue.
@Peter, thanks for testing the image and sharing your feedback.
Regarding to Victor's link to redmine, I've built and uploaded a new image which contains both kernel fixes and the latest netfilter updates which have been sent to our development mailing list.
Please test once more if your DNS and VPN issues still are existant.
Best regards,
-Stefan
peter.mueller@ipfire.org:
Hello Stefan, hello Peter, hello *,
@Stefan: Thank you again for building the ISO with Suricata 5.0.0- beta1, Rust and current libhtp.
@Peter: Sorry for having not answered your question: The problem is not only related to DNS traffic, but to new connections in general (no matter if they are encrypted or plain text) - see below for details.
Initially, Suricata refuses to start:
Sep 8 11:59:43 maverick suricata: [ERRCODE: SC_WARN_NO_STATS_LOGGERS(261)] - stats are enabled but no loggers are active Sep 8 11:59:43 maverick suricata: [ERRCODE: SC_ERR_NFQ_OPEN(68)] - no queue for given index Sep 8 11:59:43 maverick suricata: [ERRCODE: SC_ERR_NFQ_THREAD_INIT(78)] - nfq thread failed to initialize
Although statistics are disabled in /etc/suricata/suricata.yaml , enabling the statistics logger was necessary. Perhaps a glitch in the beta version, as the statistics log file is empty.
Instead of passing the NFQ indices one by one ("-q 0 -q 1 -q 2 -q 3"), Suricata now likes them as a range: "-q 0:3" After changing this in the initscript and deleting orphaned PID file, Suricata starts correctly.
Initialisation procedure takes 87 seconds on my testing hardware, which is approximately two times faster compared to Suricata 4.x. Rule parsing works, all tested attacks were successfully detected.
Resource consumption of Suricata 5.x is a bit lower compared to 4.x .
Unfortunately, both of my problems can be reproduced with that image: (a) Poor OpenVPN throughput. This has improved a bit to 1.2 MB/sec peak, but still is lower than the 2.1 MB/sec I observe on another productive machine. (b) Establishing connections and DNS resolutions takes age Regardless of SSH, HTTP, HTTPS, SMTPS or IMAPS, establishing a new connection takes 1-2 seconds due to massive packet loss. Resolving DNS records using "dig" or "host" is fast, but using "wget" or "curl" is slow.
Increasing memory allocations or "max-pending-packets" did not help again. That being said, I think we can be more generous regarding our memory allocations, as most RAM of my testing hardware stayed unallocated. :-)
As Eric Leblond already mentioned on the OISF mailing list, the actual problem seems to be something else (netfilter/iptables/?).
Version commands for reference:
[root@maverick ~]# suricata -V This is Suricata version 5.0.0-beta1 RELEASE [root@maverick ~]# uname -a Linux maverick 4.14.138-ipfire #1 SMP Sat Sep 7 06:27:36 GMT 2019 x86_64 Intel(R) Celeron(R) CPU N3150 @ 1.60GHz GenuineIntel GNU/Linux [root@maverick ~]# suricata --build-info This is Suricata version 5.0.0-beta1 RELEASE Features: NFQ PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_LIBJANSSON TLS MAGIC RUST SIMD support: none Atomic intrisics: 1 2 4 8 byte(s) 64-bits, Little-endian architecture GCC version 8.3.0, C version 199901 compiled with _FORTIFY_SOURCE=2 L1 cache line size (CLS)=64 thread local storage method: __thread compiled with LibHTP v0.5.30, linked against LibHTP v0.5.30
Suricata Configuration: AF_PACKET support: yes eBPF support: no XDP support: no PF_RING support: no NFQueue support: yes NFLOG support: no IPFW support: no Netmap support: no DAG enabled: no Napatech enabled: no WinDivert enabled: no
Unix socket enabled: yes Detection enabled: yes
Libmagic support: yes libnss support: yes libnspr support: yes libjansson support: yes liblzma support: yes hiredis support: no hiredis async with libevent: no Prelude support: no PCRE jit: yes LUA support: no libluajit: no libgeoip: no Non-bundled htp: yes Old barnyard2 support: no Hyperscan support: yes Libnet support: yes liblz4 support: no
Rust support: yes Rust strict mode: no Rust debug mode: no Rust compiler: rustc 1.37.0 (eae3437df 2019-08-13) Rust cargo: cargo 1.37.0 (9edd08916 2019-08-02)
Python support: no Python path: not set Python version: not set Python distutils no Python yaml no Install suricatactl: requires python Install suricatasc: requires python Install suricata-update: not bundled
Profiling enabled: no Profiling locks enabled: no
Development settings: Coccinelle / spatch: no Unit tests enabled: no Debug output enabled: no Debug validation enabled: no
Generic build parameters: Installation prefix: /usr Configuration directory: /etc/suricata/ Log directory: /var/log/suricata/
--prefix /usr --sysconfdir /etc --localstatedir /var --datarootdir /usr/share
Host: x86_64-pc-linux-gnu Compiler: gcc (exec name) / gcc (real) GCC Protect enabled: yes GCC march native enabled: no GCC Profile enabled: no Position Independent Executable enabled: no CFLAGS -O2 -pipe -Wall -fexceptions -fPIC -m64 -mindirect-branch=thunk -mfunction- return=thunk -mtune=generic -Wp,-D_FORTIFY_SOURCE=2 -Wp,- D_GLIBCXX_ASSERTIONS -fstack-protector-strong -I${srcdir}/../rust/gen/c-headers PCAP_CFLAGS -I/usr/include SECCFLAGS -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security
Thanks, and best regards, Peter Müller
I've forgot to post the link to the ISO:
https://people.ipfire.org/~stevee/IPFire_2.x_suricata_rust/ipfire-2.23.x86_6...
Best regards,
-Stefan
Hello Peter, Hello Victor, hello *,
@Victor, thanks for pointing to the bug at your bugtracker.
https://redmine.openinfosecfoundation.org/issues/2806#note-19
I think we are affected exactly by this issue.
@Peter, thanks for testing the image and sharing your feedback.
Regarding to Victor's link to redmine, I've built and uploaded a new image which contains both kernel fixes and the latest netfilter updates which have been sent to our development mailing list.
Please test once more if your DNS and VPN issues still are existant.
Best regards,
-Stefan
peter.mueller@ipfire.org:
Hello Stefan, hello Peter, hello *,
@Stefan: Thank you again for building the ISO with Suricata 5.0.0- beta1, Rust and current libhtp.
@Peter: Sorry for having not answered your question: The problem is not only related to DNS traffic, but to new connections in general (no matter if they are encrypted or plain text) - see below for details.
Initially, Suricata refuses to start:
Sep 8 11:59:43 maverick suricata: [ERRCODE: SC_WARN_NO_STATS_LOGGERS(261)] - stats are enabled but no loggers are active Sep 8 11:59:43 maverick suricata: [ERRCODE: SC_ERR_NFQ_OPEN(68)]
no queue for given index Sep 8 11:59:43 maverick suricata: [ERRCODE: SC_ERR_NFQ_THREAD_INIT(78)] - nfq thread failed to initialize
Although statistics are disabled in /etc/suricata/suricata.yaml , enabling the statistics logger was necessary. Perhaps a glitch in the beta version, as the statistics log file is empty.
Instead of passing the NFQ indices one by one ("-q 0 -q 1 -q 2 -q 3"), Suricata now likes them as a range: "-q 0:3" After changing this in the initscript and deleting orphaned PID file, Suricata starts correctly.
Initialisation procedure takes 87 seconds on my testing hardware, which is approximately two times faster compared to Suricata 4.x. Rule parsing works, all tested attacks were successfully detected.
Resource consumption of Suricata 5.x is a bit lower compared to 4.x .
Unfortunately, both of my problems can be reproduced with that image: (a) Poor OpenVPN throughput. This has improved a bit to 1.2 MB/sec peak, but still is lower than the 2.1 MB/sec I observe on another productive machine. (b) Establishing connections and DNS resolutions takes age Regardless of SSH, HTTP, HTTPS, SMTPS or IMAPS, establishing a new connection takes 1-2 seconds due to massive packet loss. Resolving DNS records using "dig" or "host" is fast, but using "wget" or "curl" is slow.
Increasing memory allocations or "max-pending-packets" did not help again. That being said, I think we can be more generous regarding our memory allocations, as most RAM of my testing hardware stayed unallocated. :-)
As Eric Leblond already mentioned on the OISF mailing list, the actual problem seems to be something else (netfilter/iptables/?).
Version commands for reference:
[root@maverick ~]# suricata -V This is Suricata version 5.0.0-beta1 RELEASE [root@maverick ~]# uname -a Linux maverick 4.14.138-ipfire #1 SMP Sat Sep 7 06:27:36 GMT 2019 x86_64 Intel(R) Celeron(R) CPU N3150 @ 1.60GHz GenuineIntel GNU/Linux [root@maverick ~]# suricata --build-info This is Suricata version 5.0.0-beta1 RELEASE Features: NFQ PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_LIBJANSSON TLS MAGIC RUST SIMD support: none Atomic intrisics: 1 2 4 8 byte(s) 64-bits, Little-endian architecture GCC version 8.3.0, C version 199901 compiled with _FORTIFY_SOURCE=2 L1 cache line size (CLS)=64 thread local storage method: __thread compiled with LibHTP v0.5.30, linked against LibHTP v0.5.30
Suricata Configuration: AF_PACKET support: yes eBPF support: no XDP support: no PF_RING support: no NFQueue support: yes NFLOG support: no IPFW support: no Netmap support: no DAG enabled: no Napatech enabled: no WinDivert enabled: no
Unix socket enabled: yes Detection enabled: yes
Libmagic support: yes libnss support: yes libnspr support: yes libjansson support: yes liblzma support: yes hiredis support: no hiredis async with libevent: no Prelude support: no PCRE jit: yes LUA support: no libluajit: no libgeoip: no Non-bundled htp: yes Old barnyard2 support: no Hyperscan support: yes Libnet support: yes liblz4 support: no
Rust support: yes Rust strict mode: no Rust debug mode: no Rust compiler: rustc 1.37.0 (eae3437df 2019-08-13) Rust cargo: cargo 1.37.0 (9edd08916 2019-08-02)
Python support: no Python path: not set Python version: not set Python distutils no Python yaml no Install suricatactl: requires python Install suricatasc: requires python Install suricata-update: not bundled
Profiling enabled: no Profiling locks enabled: no
Development settings: Coccinelle / spatch: no Unit tests enabled: no Debug output enabled: no Debug validation enabled: no
Generic build parameters: Installation prefix: /usr Configuration directory: /etc/suricata/ Log directory: /var/log/suricata/
--prefix /usr --sysconfdir /etc --localstatedir /var --datarootdir /usr/share
Host: x86_64-pc-linux-gnu Compiler: gcc (exec name) / gcc (real) GCC Protect enabled: yes GCC march native enabled: no GCC Profile enabled: no Position Independent Executable enabled: no CFLAGS -O2 -pipe -Wall -fexceptions -fPIC -m64 -mindirect-branch=thunk -mfunction- return=thunk -mtune=generic -Wp,-D_FORTIFY_SOURCE=2 -Wp,- D_GLIBCXX_ASSERTIONS -fstack-protector-strong -I${srcdir}/../rust/gen/c-headers PCAP_CFLAGS -I/usr/include SECCFLAGS -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security
Thanks, and best regards, Peter Müller
-
peter.mueller＠ipfire.org -
Stefan Schantl