Re: Testing report regarding image: next-suricata-rust/7808a8a2-dirty

[Stefan Schantl <stefan.schantl@ipfire.org>]

[-- Attachment #1: Type: text/plain, Size: 7824 bytes --]

Hello Peter, Hello Victor, hello *,

@Victor, thanks for pointing to the bug at your bugtracker.

https://redmine.openinfosecfoundation.org/issues/2806#note-19

I think we are affected exactly by this issue.

@Peter, thanks for testing the image and sharing your feedback.

Regarding to Victor's link to redmine, I've built and uploaded a
new image which contains both kernel fixes and the latest netfilter
updates which have been sent to our development mailing list.

Please test once more if your DNS  and VPN issues still are existant.

Best regards,

-Stefan

peter.mueller(a)ipfire.org:
> Hello Stefan, hello Peter, hello *,
> 
> @Stefan: Thank you again for building the ISO with Suricata 5.0.0-
> beta1,
> Rust and current libhtp.
> 
> @Peter: Sorry for having not answered your question: The problem is
> not
> only related to DNS traffic, but to new connections in general (no
> matter
> if they are encrypted or plain text) - see below for details.
> 
> Initially, Suricata refuses to start:
> > Sep  8 11:59:43 maverick suricata: [ERRCODE:
> > SC_WARN_NO_STATS_LOGGERS(261)] - stats are enabled but no loggers
> > are active
> > Sep  8 11:59:43 maverick suricata: [ERRCODE: SC_ERR_NFQ_OPEN(68)] -
> > no queue for given index
> > Sep  8 11:59:43 maverick suricata: [ERRCODE:
> > SC_ERR_NFQ_THREAD_INIT(78)] - nfq thread failed to initialize
> 
> Although statistics are disabled in /etc/suricata/suricata.yaml ,
> enabling
> the statistics logger was necessary. Perhaps a glitch in the beta
> version,
> as the statistics log file is empty.
> 
> Instead of passing the NFQ indices one by one ("-q 0 -q 1 -q 2 -q
> 3"),
> Suricata now likes them as a range: "-q 0:3" After changing this in
> the
> initscript and deleting orphaned PID file, Suricata starts correctly.
> 
> Initialisation procedure takes 87 seconds on my testing hardware,
> which
> is approximately two times faster compared to Suricata 4.x. Rule
> parsing
> works, all tested attacks were successfully detected.
> 
> Resource consumption of Suricata 5.x is a bit lower compared to 4.x .
> 
> Unfortunately, both of my problems can be reproduced with that image:
> (a) Poor OpenVPN throughput.
>     This has improved a bit to 1.2 MB/sec peak, but still is lower
> than
>     the 2.1 MB/sec I observe on another productive machine.
> (b) Establishing connections and DNS resolutions takes age
>     Regardless of SSH, HTTP, HTTPS, SMTPS or IMAPS, establishing a
> new
>     connection takes 1-2 seconds due to massive packet loss.
> Resolving
>     DNS records using "dig" or "host" is fast, but using "wget" or
> "curl"
>     is slow.
> 
> Increasing memory allocations or "max-pending-packets" did not help
> again.
> That being said, I think we can be more generous regarding our memory
> allocations, as most RAM of my testing hardware stayed unallocated.
> :-)
> 
> As Eric Leblond already mentioned on the OISF mailing list, the
> actual
> problem seems to be something else (netfilter/iptables/?).
> 
> Version commands for reference:
> 
> > [root(a)maverick ~]# suricata -V
> > This is Suricata version 5.0.0-beta1 RELEASE
> > [root(a)maverick ~]# uname -a
> > Linux maverick 4.14.138-ipfire #1 SMP Sat Sep 7 06:27:36 GMT 2019
> > x86_64 Intel(R) Celeron(R) CPU N3150 @ 1.60GHz GenuineIntel
> > GNU/Linux
> > [root(a)maverick ~]# suricata --build-info
> > This is Suricata version 5.0.0-beta1 RELEASE
> > Features: NFQ PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG
> > LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_LIBJANSSON TLS
> > MAGIC RUST 
> > SIMD support: none
> > Atomic intrisics: 1 2 4 8 byte(s)
> > 64-bits, Little-endian architecture
> > GCC version 8.3.0, C version 199901
> > compiled with _FORTIFY_SOURCE=2
> > L1 cache line size (CLS)=64
> > thread local storage method: __thread
> > compiled with LibHTP v0.5.30, linked against LibHTP v0.5.30
> > 
> > Suricata Configuration:
> >   AF_PACKET support:                       yes
> >   eBPF support:                            no
> >   XDP support:                             no
> >   PF_RING support:                         no
> >   NFQueue support:                         yes
> >   NFLOG support:                           no
> >   IPFW support:                            no
> >   Netmap support:                          no 
> >   DAG enabled:                             no
> >   Napatech enabled:                        no
> >   WinDivert enabled:                       no
> > 
> >   Unix socket enabled:                     yes
> >   Detection enabled:                       yes
> > 
> >   Libmagic support:                        yes
> >   libnss support:                          yes
> >   libnspr support:                         yes
> >   libjansson support:                      yes
> >   liblzma support:                         yes
> >   hiredis support:                         no
> >   hiredis async with libevent:             no
> >   Prelude support:                         no
> >   PCRE jit:                                yes
> >   LUA support:                             no
> >   libluajit:                               no
> >   libgeoip:                                no
> >   Non-bundled htp:                         yes
> >   Old barnyard2 support:                   no
> >   Hyperscan support:                       yes
> >   Libnet support:                          yes
> >   liblz4 support:                          no
> > 
> >   Rust support:                            yes
> >   Rust strict mode:                        no
> >   Rust debug mode:                         no
> >   Rust compiler:                           rustc 1.37.0 (eae3437df
> > 2019-08-13)
> >   Rust cargo:                              cargo 1.37.0 (9edd08916
> > 2019-08-02)
> > 
> >   Python support:                          no
> >   Python path:                             not set
> >   Python version:                          not set
> >   Python distutils                         no
> >   Python yaml                              no
> >   Install suricatactl:                     requires python
> >   Install suricatasc:                      requires python
> >   Install suricata-update:                 not bundled
> > 
> >   Profiling enabled:                       no
> >   Profiling locks enabled:                 no
> > 
> > Development settings:
> >   Coccinelle / spatch:                     no
> >   Unit tests enabled:                      no
> >   Debug output enabled:                    no
> >   Debug validation enabled:                no
> > 
> > Generic build parameters:
> >   Installation prefix:                     /usr
> >   Configuration directory:                 /etc/suricata/
> >   Log directory:                           /var/log/suricata/
> > 
> >   --prefix                                 /usr
> >   --sysconfdir                             /etc
> >   --localstatedir                          /var
> >   --datarootdir                            /usr/share
> > 
> >   Host:                                    x86_64-pc-linux-gnu
> >   Compiler:                                gcc (exec name) / gcc
> > (real)
> >   GCC Protect enabled:                     yes
> >   GCC march native enabled:                no
> >   GCC Profile enabled:                     no
> >   Position Independent Executable enabled: no
> >   CFLAGS                                   -O2 -pipe -Wall
> > -fexceptions -fPIC -m64 -mindirect-branch=thunk -mfunction-
> > return=thunk -mtune=generic -Wp,-D_FORTIFY_SOURCE=2 -Wp,-
> > D_GLIBCXX_ASSERTIONS -fstack-protector-strong
> > -I${srcdir}/../rust/gen/c-headers
> >   PCAP_CFLAGS                               -I/usr/include
> >   SECCFLAGS                                -fstack-protector
> > -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security
> 
> Thanks, and best regards,
> Peter Müller