From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Tremer To: development@lists.ipfire.org Subject: Re: Peeking at unbound statistics from WUI Date: Tue, 22 Jan 2019 14:16:43 +0000 Message-ID: <8DC5B693-6FB2-445F-A4DB-F24A471A5D11@ipfire.org> In-Reply-To: =?utf-8?q?=3CAM0PR03MB493209C3298947E18CBBD66DB69F0=40AM0PR03MB?= =?utf-8?q?4932=2Eeurprd03=2Eprod=2Eoutlook=2Ecom=3E?= MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============7747541465294769672==" List-Id: --===============7747541465294769672== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hey, > On 21 Jan 2019, at 19:54, Rachid Groeneveld = wrote: >=20 > Hi Michael, > =20 > >Yes, they are quite pretty to look at. I am not entirely sure why people a= re so much in love with them because many dashboards out there (not talking a= bout anything specifically) do not transport much information. They just show= that the =E2=80=9Csystem is busy=E2=80=9D. I am not sure whether that is use= ful information or not. > >=20 > >IPFire doesn=E2=80=99t have a very pretty stats section, but it does the j= ob. The main job is to show what has happened over time. That is in recent hi= story like =E2=80=9Clast week=E2=80=9D or =E2=80=9Cyesterday=E2=80=9D so that= problems with the system can be spotted very easily; and showing trends over= time so that people can see if bandwidth usage has =E2=80=9Cdoubled=E2=80=9D= over a year or so. That would be information I care about. > >=20 > >We have a thing called collecty which is supposed to achieve that in IPFir= e 3. It collects various system statistics, but you can probably add unbound = and some other sources, too. It is written in Python which makes it easier to= extend - although it has a little C module to talk to hard drives. It potent= ially could be ported to IPFire 2. > >=20 > >Maybe it is worth having a look at that. IPFire 2 right now has loads of d= ifferent scripts and tools that collect statistics. It would be nice to have = this all in the same place and use it where ever we want to present this data. > >=20 > >Here is the code: > >=20 > > https://git.ipfire.org/?p=3Doddments/collecty.git;a=3Dsummary > >=20 > >This is an example graph in SVG: > >=20 > > https://people.ipfire.org/~ms/graph.svg > >=20 > >Right now, there is no mechanism to get the raw data, but that is trivial = to add. We could store the last hour with a really high resolution to make JS= graphs looks nice, too. > >=20 > >-Michael > =20 > I'm not saying current graphs don't do their job, I just wanted to integrat= e the graphs in the theme nicely, and in order to do so I need raw data; so I= can have the graphs drawn by the framework. > This also means it'll be fairly easy to save lots of data (in a relational = database), which can be queried easily. Then you can show trends over years, = because saving a few numbers in a database is not very expensive in terms of = performance and storage. A relational database is probably not the right tool here. That highly depend= s on what you want to achieve though. With my two objectives outlined earlier, you will have a massive amount of da= ta and the older the data gets, the more irrelevant it is wasting loads of sp= ace. Space was a huge issue here in the past. Many firewalls ran on embedded = systems with slow and expensive flash storage which did not survive too many = cycles. The solution that we have that is really great is an RRD database: https://oss.oetiker.ch/rrdtool/index.en.html It stores values with different precision and generates averages over time so= that historical data is still present, but with less resolution. That gives = you a database of constant and compact size. It=E2=80=99s a cool thing. =20 > I agree on the single stats gathering entity, this makes it easier to maint= ain. The reason I'm so interested in the WUI, is that I want to be able to se= e what's happening on the network and be able to respond to that. For example= if I'm seeing lots of egression to countries I don't trust, I want to be inf= ormed (automatically) and confirm the behavior in the WUI. As for DNS, if som= eone is visiting lots of untrustworthy sites, I'd like to be able to see that= . Maybe I'm just curious out of bounds, that's possible too =F0=9F=98=89 Plus= I like to tinker, unbound stats was just a quick way of getting DNS stats, t= he dnstap plugin logs way more (async, so it shouldn't affect performance too= much). That info can be stored in a database and queried whenever. I am not sure if DNS will tell you where your packets are going... > Finally there's the question of out-of-date information, is a cron job freq= uent enough to manage the network or should on-demand info be acquired for re= search? It all depends on the way the data, or information if you will, is go= ing to be used. > =20 > I just thought DNS stats would be the easiest to gather, since the command = provided me with simple stats. What stats and information is required for bus= iness still requires research, but since pihole stats were so popular I thoug= ht it would make for a good start. But I think firewall and IDS stats would b= e even more interesting for people to see/monitor. > =20 > Anyways, just thinking out loud here, from a SIEM point of view it seemed r= easonable to also investigate DNS behavior. =F0=9F=98=8A I will have a look a= t the collect solution and c-binary one. I like thinking out loud :) > =20 > Thanks for all the pointers! > =20 > Cheers, > =20 > Rachid --===============7747541465294769672==--