* Testing report regarding image: next-suricata-rust/7808a8a2-dirty
@ 2019-09-08 10:55 peter.mueller
2019-09-09 10:34 ` Stefan Schantl
0 siblings, 1 reply; 3+ messages in thread
From: peter.mueller @ 2019-09-08 10:55 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 6992 bytes --]
Hello Stefan, hello Peter, hello *,
@Stefan: Thank you again for building the ISO with Suricata 5.0.0-beta1,
Rust and current libhtp.
@Peter: Sorry for having not answered your question: The problem is not
only related to DNS traffic, but to new connections in general (no matter
if they are encrypted or plain text) - see below for details.
Initially, Suricata refuses to start:
> Sep 8 11:59:43 maverick suricata: [ERRCODE: SC_WARN_NO_STATS_LOGGERS(261)] - stats are enabled but no loggers are active
> Sep 8 11:59:43 maverick suricata: [ERRCODE: SC_ERR_NFQ_OPEN(68)] - no queue for given index
> Sep 8 11:59:43 maverick suricata: [ERRCODE: SC_ERR_NFQ_THREAD_INIT(78)] - nfq thread failed to initialize
Although statistics are disabled in /etc/suricata/suricata.yaml , enabling
the statistics logger was necessary. Perhaps a glitch in the beta version,
as the statistics log file is empty.
Instead of passing the NFQ indices one by one ("-q 0 -q 1 -q 2 -q 3"),
Suricata now likes them as a range: "-q 0:3" After changing this in the
initscript and deleting orphaned PID file, Suricata starts correctly.
Initialisation procedure takes 87 seconds on my testing hardware, which
is approximately two times faster compared to Suricata 4.x. Rule parsing
works, all tested attacks were successfully detected.
Resource consumption of Suricata 5.x is a bit lower compared to 4.x .
Unfortunately, both of my problems can be reproduced with that image:
(a) Poor OpenVPN throughput.
This has improved a bit to 1.2 MB/sec peak, but still is lower than
the 2.1 MB/sec I observe on another productive machine.
(b) Establishing connections and DNS resolutions takes age
Regardless of SSH, HTTP, HTTPS, SMTPS or IMAPS, establishing a new
connection takes 1-2 seconds due to massive packet loss. Resolving
DNS records using "dig" or "host" is fast, but using "wget" or "curl"
is slow.
Increasing memory allocations or "max-pending-packets" did not help again.
That being said, I think we can be more generous regarding our memory
allocations, as most RAM of my testing hardware stayed unallocated. :-)
As Eric Leblond already mentioned on the OISF mailing list, the actual
problem seems to be something else (netfilter/iptables/?).
Version commands for reference:
> [root(a)maverick ~]# suricata -V
> This is Suricata version 5.0.0-beta1 RELEASE
> [root(a)maverick ~]# uname -a
> Linux maverick 4.14.138-ipfire #1 SMP Sat Sep 7 06:27:36 GMT 2019 x86_64 Intel(R) Celeron(R) CPU N3150 @ 1.60GHz GenuineIntel GNU/Linux
> [root(a)maverick ~]# suricata --build-info
> This is Suricata version 5.0.0-beta1 RELEASE
> Features: NFQ PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_LIBJANSSON TLS MAGIC RUST
> SIMD support: none
> Atomic intrisics: 1 2 4 8 byte(s)
> 64-bits, Little-endian architecture
> GCC version 8.3.0, C version 199901
> compiled with _FORTIFY_SOURCE=2
> L1 cache line size (CLS)=64
> thread local storage method: __thread
> compiled with LibHTP v0.5.30, linked against LibHTP v0.5.30
>
> Suricata Configuration:
> AF_PACKET support: yes
> eBPF support: no
> XDP support: no
> PF_RING support: no
> NFQueue support: yes
> NFLOG support: no
> IPFW support: no
> Netmap support: no
> DAG enabled: no
> Napatech enabled: no
> WinDivert enabled: no
>
> Unix socket enabled: yes
> Detection enabled: yes
>
> Libmagic support: yes
> libnss support: yes
> libnspr support: yes
> libjansson support: yes
> liblzma support: yes
> hiredis support: no
> hiredis async with libevent: no
> Prelude support: no
> PCRE jit: yes
> LUA support: no
> libluajit: no
> libgeoip: no
> Non-bundled htp: yes
> Old barnyard2 support: no
> Hyperscan support: yes
> Libnet support: yes
> liblz4 support: no
>
> Rust support: yes
> Rust strict mode: no
> Rust debug mode: no
> Rust compiler: rustc 1.37.0 (eae3437df 2019-08-13)
> Rust cargo: cargo 1.37.0 (9edd08916 2019-08-02)
>
> Python support: no
> Python path: not set
> Python version: not set
> Python distutils no
> Python yaml no
> Install suricatactl: requires python
> Install suricatasc: requires python
> Install suricata-update: not bundled
>
> Profiling enabled: no
> Profiling locks enabled: no
>
> Development settings:
> Coccinelle / spatch: no
> Unit tests enabled: no
> Debug output enabled: no
> Debug validation enabled: no
>
> Generic build parameters:
> Installation prefix: /usr
> Configuration directory: /etc/suricata/
> Log directory: /var/log/suricata/
>
> --prefix /usr
> --sysconfdir /etc
> --localstatedir /var
> --datarootdir /usr/share
>
> Host: x86_64-pc-linux-gnu
> Compiler: gcc (exec name) / gcc (real)
> GCC Protect enabled: yes
> GCC march native enabled: no
> GCC Profile enabled: no
> Position Independent Executable enabled: no
> CFLAGS -O2 -pipe -Wall -fexceptions -fPIC -m64 -mindirect-branch=thunk -mfunction-return=thunk -mtune=generic -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -I${srcdir}/../rust/gen/c-headers
> PCAP_CFLAGS -I/usr/include
> SECCFLAGS -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security
Thanks, and best regards,
Peter Müller
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Testing report regarding image: next-suricata-rust/7808a8a2-dirty
2019-09-08 10:55 Testing report regarding image: next-suricata-rust/7808a8a2-dirty peter.mueller
@ 2019-09-09 10:34 ` Stefan Schantl
2019-09-09 10:53 ` Stefan Schantl
0 siblings, 1 reply; 3+ messages in thread
From: Stefan Schantl @ 2019-09-09 10:34 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 7824 bytes --]
Hello Peter, Hello Victor, hello *,
@Victor, thanks for pointing to the bug at your bugtracker.
https://redmine.openinfosecfoundation.org/issues/2806#note-19
I think we are affected exactly by this issue.
@Peter, thanks for testing the image and sharing your feedback.
Regarding to Victor's link to redmine, I've built and uploaded a
new image which contains both kernel fixes and the latest netfilter
updates which have been sent to our development mailing list.
Please test once more if your DNS and VPN issues still are existant.
Best regards,
-Stefan
peter.mueller(a)ipfire.org:
> Hello Stefan, hello Peter, hello *,
>
> @Stefan: Thank you again for building the ISO with Suricata 5.0.0-
> beta1,
> Rust and current libhtp.
>
> @Peter: Sorry for having not answered your question: The problem is
> not
> only related to DNS traffic, but to new connections in general (no
> matter
> if they are encrypted or plain text) - see below for details.
>
> Initially, Suricata refuses to start:
> > Sep 8 11:59:43 maverick suricata: [ERRCODE:
> > SC_WARN_NO_STATS_LOGGERS(261)] - stats are enabled but no loggers
> > are active
> > Sep 8 11:59:43 maverick suricata: [ERRCODE: SC_ERR_NFQ_OPEN(68)] -
> > no queue for given index
> > Sep 8 11:59:43 maverick suricata: [ERRCODE:
> > SC_ERR_NFQ_THREAD_INIT(78)] - nfq thread failed to initialize
>
> Although statistics are disabled in /etc/suricata/suricata.yaml ,
> enabling
> the statistics logger was necessary. Perhaps a glitch in the beta
> version,
> as the statistics log file is empty.
>
> Instead of passing the NFQ indices one by one ("-q 0 -q 1 -q 2 -q
> 3"),
> Suricata now likes them as a range: "-q 0:3" After changing this in
> the
> initscript and deleting orphaned PID file, Suricata starts correctly.
>
> Initialisation procedure takes 87 seconds on my testing hardware,
> which
> is approximately two times faster compared to Suricata 4.x. Rule
> parsing
> works, all tested attacks were successfully detected.
>
> Resource consumption of Suricata 5.x is a bit lower compared to 4.x .
>
> Unfortunately, both of my problems can be reproduced with that image:
> (a) Poor OpenVPN throughput.
> This has improved a bit to 1.2 MB/sec peak, but still is lower
> than
> the 2.1 MB/sec I observe on another productive machine.
> (b) Establishing connections and DNS resolutions takes age
> Regardless of SSH, HTTP, HTTPS, SMTPS or IMAPS, establishing a
> new
> connection takes 1-2 seconds due to massive packet loss.
> Resolving
> DNS records using "dig" or "host" is fast, but using "wget" or
> "curl"
> is slow.
>
> Increasing memory allocations or "max-pending-packets" did not help
> again.
> That being said, I think we can be more generous regarding our memory
> allocations, as most RAM of my testing hardware stayed unallocated.
> :-)
>
> As Eric Leblond already mentioned on the OISF mailing list, the
> actual
> problem seems to be something else (netfilter/iptables/?).
>
> Version commands for reference:
>
> > [root(a)maverick ~]# suricata -V
> > This is Suricata version 5.0.0-beta1 RELEASE
> > [root(a)maverick ~]# uname -a
> > Linux maverick 4.14.138-ipfire #1 SMP Sat Sep 7 06:27:36 GMT 2019
> > x86_64 Intel(R) Celeron(R) CPU N3150 @ 1.60GHz GenuineIntel
> > GNU/Linux
> > [root(a)maverick ~]# suricata --build-info
> > This is Suricata version 5.0.0-beta1 RELEASE
> > Features: NFQ PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG
> > LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_LIBJANSSON TLS
> > MAGIC RUST
> > SIMD support: none
> > Atomic intrisics: 1 2 4 8 byte(s)
> > 64-bits, Little-endian architecture
> > GCC version 8.3.0, C version 199901
> > compiled with _FORTIFY_SOURCE=2
> > L1 cache line size (CLS)=64
> > thread local storage method: __thread
> > compiled with LibHTP v0.5.30, linked against LibHTP v0.5.30
> >
> > Suricata Configuration:
> > AF_PACKET support: yes
> > eBPF support: no
> > XDP support: no
> > PF_RING support: no
> > NFQueue support: yes
> > NFLOG support: no
> > IPFW support: no
> > Netmap support: no
> > DAG enabled: no
> > Napatech enabled: no
> > WinDivert enabled: no
> >
> > Unix socket enabled: yes
> > Detection enabled: yes
> >
> > Libmagic support: yes
> > libnss support: yes
> > libnspr support: yes
> > libjansson support: yes
> > liblzma support: yes
> > hiredis support: no
> > hiredis async with libevent: no
> > Prelude support: no
> > PCRE jit: yes
> > LUA support: no
> > libluajit: no
> > libgeoip: no
> > Non-bundled htp: yes
> > Old barnyard2 support: no
> > Hyperscan support: yes
> > Libnet support: yes
> > liblz4 support: no
> >
> > Rust support: yes
> > Rust strict mode: no
> > Rust debug mode: no
> > Rust compiler: rustc 1.37.0 (eae3437df
> > 2019-08-13)
> > Rust cargo: cargo 1.37.0 (9edd08916
> > 2019-08-02)
> >
> > Python support: no
> > Python path: not set
> > Python version: not set
> > Python distutils no
> > Python yaml no
> > Install suricatactl: requires python
> > Install suricatasc: requires python
> > Install suricata-update: not bundled
> >
> > Profiling enabled: no
> > Profiling locks enabled: no
> >
> > Development settings:
> > Coccinelle / spatch: no
> > Unit tests enabled: no
> > Debug output enabled: no
> > Debug validation enabled: no
> >
> > Generic build parameters:
> > Installation prefix: /usr
> > Configuration directory: /etc/suricata/
> > Log directory: /var/log/suricata/
> >
> > --prefix /usr
> > --sysconfdir /etc
> > --localstatedir /var
> > --datarootdir /usr/share
> >
> > Host: x86_64-pc-linux-gnu
> > Compiler: gcc (exec name) / gcc
> > (real)
> > GCC Protect enabled: yes
> > GCC march native enabled: no
> > GCC Profile enabled: no
> > Position Independent Executable enabled: no
> > CFLAGS -O2 -pipe -Wall
> > -fexceptions -fPIC -m64 -mindirect-branch=thunk -mfunction-
> > return=thunk -mtune=generic -Wp,-D_FORTIFY_SOURCE=2 -Wp,-
> > D_GLIBCXX_ASSERTIONS -fstack-protector-strong
> > -I${srcdir}/../rust/gen/c-headers
> > PCAP_CFLAGS -I/usr/include
> > SECCFLAGS -fstack-protector
> > -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security
>
> Thanks, and best regards,
> Peter Müller
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Testing report regarding image: next-suricata-rust/7808a8a2-dirty
2019-09-09 10:34 ` Stefan Schantl
@ 2019-09-09 10:53 ` Stefan Schantl
0 siblings, 0 replies; 3+ messages in thread
From: Stefan Schantl @ 2019-09-09 10:53 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 8636 bytes --]
I've forgot to post the link to the ISO:
https://people.ipfire.org/~stevee/IPFire_2.x_suricata_rust/ipfire-2.23.x86_64-full-core136.iso
Best regards,
-Stefan
> Hello Peter, Hello Victor, hello *,
>
> @Victor, thanks for pointing to the bug at your bugtracker.
>
> https://redmine.openinfosecfoundation.org/issues/2806#note-19
>
> I think we are affected exactly by this issue.
>
> @Peter, thanks for testing the image and sharing your feedback.
>
> Regarding to Victor's link to redmine, I've built and uploaded a
> new image which contains both kernel fixes and the latest netfilter
> updates which have been sent to our development mailing list.
>
> Please test once more if your DNS and VPN issues still are existant.
>
> Best regards,
>
> -Stefan
>
> peter.mueller(a)ipfire.org:
> > Hello Stefan, hello Peter, hello *,
> >
> > @Stefan: Thank you again for building the ISO with Suricata 5.0.0-
> > beta1,
> > Rust and current libhtp.
> >
> > @Peter: Sorry for having not answered your question: The problem is
> > not
> > only related to DNS traffic, but to new connections in general (no
> > matter
> > if they are encrypted or plain text) - see below for details.
> >
> > Initially, Suricata refuses to start:
> > > Sep 8 11:59:43 maverick suricata: [ERRCODE:
> > > SC_WARN_NO_STATS_LOGGERS(261)] - stats are enabled but no loggers
> > > are active
> > > Sep 8 11:59:43 maverick suricata: [ERRCODE: SC_ERR_NFQ_OPEN(68)]
> > > -
> > > no queue for given index
> > > Sep 8 11:59:43 maverick suricata: [ERRCODE:
> > > SC_ERR_NFQ_THREAD_INIT(78)] - nfq thread failed to initialize
> >
> > Although statistics are disabled in /etc/suricata/suricata.yaml ,
> > enabling
> > the statistics logger was necessary. Perhaps a glitch in the beta
> > version,
> > as the statistics log file is empty.
> >
> > Instead of passing the NFQ indices one by one ("-q 0 -q 1 -q 2 -q
> > 3"),
> > Suricata now likes them as a range: "-q 0:3" After changing this in
> > the
> > initscript and deleting orphaned PID file, Suricata starts
> > correctly.
> >
> > Initialisation procedure takes 87 seconds on my testing hardware,
> > which
> > is approximately two times faster compared to Suricata 4.x. Rule
> > parsing
> > works, all tested attacks were successfully detected.
> >
> > Resource consumption of Suricata 5.x is a bit lower compared to 4.x
> > .
> >
> > Unfortunately, both of my problems can be reproduced with that
> > image:
> > (a) Poor OpenVPN throughput.
> > This has improved a bit to 1.2 MB/sec peak, but still is lower
> > than
> > the 2.1 MB/sec I observe on another productive machine.
> > (b) Establishing connections and DNS resolutions takes age
> > Regardless of SSH, HTTP, HTTPS, SMTPS or IMAPS, establishing a
> > new
> > connection takes 1-2 seconds due to massive packet loss.
> > Resolving
> > DNS records using "dig" or "host" is fast, but using "wget" or
> > "curl"
> > is slow.
> >
> > Increasing memory allocations or "max-pending-packets" did not help
> > again.
> > That being said, I think we can be more generous regarding our
> > memory
> > allocations, as most RAM of my testing hardware stayed unallocated.
> > :-)
> >
> > As Eric Leblond already mentioned on the OISF mailing list, the
> > actual
> > problem seems to be something else (netfilter/iptables/?).
> >
> > Version commands for reference:
> >
> > > [root(a)maverick ~]# suricata -V
> > > This is Suricata version 5.0.0-beta1 RELEASE
> > > [root(a)maverick ~]# uname -a
> > > Linux maverick 4.14.138-ipfire #1 SMP Sat Sep 7 06:27:36 GMT 2019
> > > x86_64 Intel(R) Celeron(R) CPU N3150 @ 1.60GHz GenuineIntel
> > > GNU/Linux
> > > [root(a)maverick ~]# suricata --build-info
> > > This is Suricata version 5.0.0-beta1 RELEASE
> > > Features: NFQ PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT
> > > LIBCAP_NG
> > > LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_LIBJANSSON
> > > TLS
> > > MAGIC RUST
> > > SIMD support: none
> > > Atomic intrisics: 1 2 4 8 byte(s)
> > > 64-bits, Little-endian architecture
> > > GCC version 8.3.0, C version 199901
> > > compiled with _FORTIFY_SOURCE=2
> > > L1 cache line size (CLS)=64
> > > thread local storage method: __thread
> > > compiled with LibHTP v0.5.30, linked against LibHTP v0.5.30
> > >
> > > Suricata Configuration:
> > > AF_PACKET support: yes
> > > eBPF support: no
> > > XDP support: no
> > > PF_RING support: no
> > > NFQueue support: yes
> > > NFLOG support: no
> > > IPFW support: no
> > > Netmap support: no
> > > DAG enabled: no
> > > Napatech enabled: no
> > > WinDivert enabled: no
> > >
> > > Unix socket enabled: yes
> > > Detection enabled: yes
> > >
> > > Libmagic support: yes
> > > libnss support: yes
> > > libnspr support: yes
> > > libjansson support: yes
> > > liblzma support: yes
> > > hiredis support: no
> > > hiredis async with libevent: no
> > > Prelude support: no
> > > PCRE jit: yes
> > > LUA support: no
> > > libluajit: no
> > > libgeoip: no
> > > Non-bundled htp: yes
> > > Old barnyard2 support: no
> > > Hyperscan support: yes
> > > Libnet support: yes
> > > liblz4 support: no
> > >
> > > Rust support: yes
> > > Rust strict mode: no
> > > Rust debug mode: no
> > > Rust compiler: rustc 1.37.0
> > > (eae3437df
> > > 2019-08-13)
> > > Rust cargo: cargo 1.37.0
> > > (9edd08916
> > > 2019-08-02)
> > >
> > > Python support: no
> > > Python path: not set
> > > Python version: not set
> > > Python distutils no
> > > Python yaml no
> > > Install suricatactl: requires python
> > > Install suricatasc: requires python
> > > Install suricata-update: not bundled
> > >
> > > Profiling enabled: no
> > > Profiling locks enabled: no
> > >
> > > Development settings:
> > > Coccinelle / spatch: no
> > > Unit tests enabled: no
> > > Debug output enabled: no
> > > Debug validation enabled: no
> > >
> > > Generic build parameters:
> > > Installation prefix: /usr
> > > Configuration directory: /etc/suricata/
> > > Log directory: /var/log/suricata/
> > >
> > > --prefix /usr
> > > --sysconfdir /etc
> > > --localstatedir /var
> > > --datarootdir /usr/share
> > >
> > > Host: x86_64-pc-linux-gnu
> > > Compiler: gcc (exec name) / gcc
> > > (real)
> > > GCC Protect enabled: yes
> > > GCC march native enabled: no
> > > GCC Profile enabled: no
> > > Position Independent Executable enabled: no
> > > CFLAGS -O2 -pipe -Wall
> > > -fexceptions -fPIC -m64 -mindirect-branch=thunk -mfunction-
> > > return=thunk -mtune=generic -Wp,-D_FORTIFY_SOURCE=2 -Wp,-
> > > D_GLIBCXX_ASSERTIONS -fstack-protector-strong
> > > -I${srcdir}/../rust/gen/c-headers
> > > PCAP_CFLAGS -I/usr/include
> > > SECCFLAGS -fstack-protector
> > > -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security
> >
> > Thanks, and best regards,
> > Peter Müller
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2019-09-09 10:53 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-09-08 10:55 Testing report regarding image: next-suricata-rust/7808a8a2-dirty peter.mueller
2019-09-09 10:34 ` Stefan Schantl
2019-09-09 10:53 ` Stefan Schantl
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox