From mboxrd@z Thu Jan 1 00:00:00 1970 From: Stefan Schantl To: development@lists.ipfire.org Subject: Re: IPFire meets Suricata - Call for tester Date: Tue, 25 Dec 2018 20:17:24 +0100 Message-ID: <29f9e44b119b2637f71fd4436514394f46a071e3.camel@ipfire.org> In-Reply-To: <7e6e3633-58bf-cbcc-9c8c-918dbd9e0838@tfitzgeorge.me.uk> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2690295716031859180==" List-Id: --===============2690295716031859180== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello Tim, thanks for joining and providing your feedback on this list. > Hi, >=20 > I've loaded a system up with the test image. I've not had much time > to > actually use it (hopefully after Christmas). I would like to know > when > there'll be an image with the marking fixed. I'll generate and upload a new one very soon. >=20 > I agree with Michael over the GUI; splitting the system settings and > ruleset settings seems like a good idea. The thing that caught me > originally was that I selected a ruleset and then clicked on > 'Download > new ruleset' and nothing happened because I hadn't clicked on 'Save'. I splitted the section into two parts. One for configuring the IDS and one which contains the rules settings for now. >=20 > A couple of points on automatic updates: >=20 > 1 - The updates shouldn't run exactly on the hour since this means > the > load on the servers will be very bursty. It would be better to > spread > the load by running at some point past the hour and ensuring that > this > is different for every system (generate a random offset during > installation?). Since the rulesets are sizeable (100 MiB for Talos > VRT) > and they're given away for free, I think it's a good idea to do > something about this. Thanks for that, I don't know if "fcron" does any kind of randomization. I'll have a closer look on it. I'm reading you are using the ruleset provided by the Talos team. I'm intrested in adding support for this. Do you have some more details about this ruleset and a download URL? >=20 > 2 - There's a warning in the oinkmaster documentation: >=20 > Oinkmaster can be run manually or automatically as a cron job. But > of=20 > course, be very careful when doing the latter. Things could easily > get=20 > messed up from one update to another because of different layout in > the=20 > rules archive, typo in a rule, bugs in the script, and so on. At > least=20 > run snort -T on the new rules before restarting your Snort process. >=20 > Since this is about the rulefiles I'm sure it applies to Suricata as > well. I have actually seen one occasion when the updated rules > failed > snort -T with my updater - and was after the download passed an > md5sum > check. So the downloaded ruleset can be corrupt, if not very often.=20 > Should probably do something like: >=20 > download update > backup old rules > run oinkmaster > if (suricata -T is OK) then > tell suricata to re-read rules > else > restore backup > end if Thanks for the hint. I have to dig into this, if suricata offers a mechanism to check the ruleset before doing a reload of it. >=20 > Otherwise, I'm encouraged by the reduction in memory usage over > snort, > although it'll be interesting to see how this holds up with a larger > ruleset and under heavier load. Best regards, -Stefan >=20 > Regards, > Tim >=20 > On 20/12/2018 14:05, Michael Tremer wrote: > > Hi, > >=20 > > > On 20 Dec 2018, at 13:03, Stefan Schantl < > > > stefan.schantl(a)ipfire.org> wrote: > > >=20 > > > Hello Michael, > > >=20 > > > thanks for having a look at suricata and your feedback. > > > > Hi, > > > >=20 > > > > So I finally had a closer look at this, too. I have a couple of > > > > questions and would like to propose some changes to the UI: > > > >=20 > > > > Questions: > > > >=20 > > > > * Is it a good idea to have =E2=80=9CNo=E2=80=9D rules as default and= also =E2=80=9Cno=E2=80=9D > > > > automatic rule updates? I guess we can configure at least > > > > Emerging > > > > Threats as a good default because they are free. I also think > > > > that > > > > when IDS is enabled, the system should update the ruleset by > > > > default > > > > once a day. When IDS is enabled, there should be no rule > > > > update. > > > >=20 > > > > * That also leads me to: Should no automatic updates even be an > > > > option? > > > >=20 > > > Making the free ruleset of emergingthreads as default is okay for > > > me - > > > ACK. > > >=20 > > > I'm also fine with doing a weekly ruleset update as default > > > setting. > > >=20 > > > My personal opinion is, it should be possible to disable the > > > automatic > > > rule update. There are some situations where this feature could > > > be > > > quite usefull, for example if you have only volume limited > > > internet > > > access or even other (which we currently do not have in > > > mind....). > > I guess if you don=E2=80=99t update the rules there is no point in running > > the IPS. It=E2=80=99s just like a virus scanner with outdated signatures: > > detecting even less. > >=20 > > Do we have more people with an opinion on this? > >=20 > > The rulesets are indeed quite large. But the only people who might > > want to disable it are people who run IPFire on an IoT appliance > > and those are too slow to run the IPS anyways. So it=E2=80=99s not a very > > strong argument in my opinion. > >=20 > > The best argument I can make for it is only that this is probably > > unwanted behaviour and therefore can easily be misconfugured. > >=20 > > > > Bugs: > > > >=20 > > > > * When I enabled IDS in IPS mode, I checked the box to activate > > > > it > > > > and selected the ET ruleset and hit save. Suricata starts but > > > > without > > > > rules which is definitely not what I wanted. Either the box > > > > should be > > > > greyed out or the downloader is launched when there are no > > > > rules, > > > > yet. > > > >=20 > > > > * It is possible to enable suricata without selecting any > > > > interfaces. > > > > That just wastes RAM and makes it easy to enable it without > > > > actually > > > > having the protection. > > > >=20 > > > > If you agree, I will create tickets for this. > > > Thanks, fixed. > > >=20 > > > https://git.ipfire.org/?p=3Dpeople/stevee/ipfire-2.x.git;a=3Dcommit;h= =3Debdd0f9a90da800cc6173f6f30fb0621dddc354b > > I am on a plane now, so I cannot see this, but thanks! > >=20 > > > > UI Stuff: > > > >=20 > > > > * I think that the settings to enable suricata up to and > > > > including > > > > =E2=80=9CTraffic analysing=E2=80=9D should be in the top box. The sec= ond box > > > > should > > > > only care about the rules. That might even solve the problem > > > > described above. > > > During development I had a very similar idea, of grouping all > > > suricata=20 > > > settings and all which are for the ruleset together into seperate > > > boxes. I dropped this idea, because of some big restrictions of > > > the > > > code and I was afraid of how many work this would be to re-write > > > huge > > > parts of it to get the desired result. > > I think that this is very little work. > >=20 > > > > * I find it quite confusing that one has to =E2=80=9CActivate Intrusi= on > > > > Detection System=E2=80=9D and then again select IDS or IPS. It doesn= =E2=80=99t > > > > seem > > > > to make much sense even to select the top box when IPS is > > > > checked. I > > > > would therefore prefer the following: one box that just says > > > > =E2=80=9CActivate=E2=80=9D and that switches on or off the whole thin= g, and > > > > then a > > > > second box that says =E2=80=9Cmonitor only=E2=80=9D or something alon= g these > > > > lines. > > > > That box should be disabled by default which will run suricata > > > > in IPS > > > > mode (which is what everybody wants!) and when checked will > > > > downgrade > > > > it to IDS mode. Right now it looks like this is an equal level > > > > of > > > > both options but I think that there is a risk here that people > > > > enable > > > > IDS and think that they are protected when they are in fact > > > > not. I > > > > think the =E2=80=9Cmonitor only=E2=80=9D checkbox makes this a lot cl= earer. > > > Fair enough - I'm going now to rework the current section for > > > choosing > > > the runmode and move from the radio buttons to a select box which > > > will > > > be called "Only monitor traffic" as you suggested and will place > > > it > > > right next to the box where you can enable the whole IPS/IDS > > > system. > > I think the best wording is: Monitor traffic only > >=20 > > > > * The Traffic Analyzing headline doesn=E2=80=99t make any sense. It > > > > should > > > > rather be =E2=80=9CMonitored zones=E2=80=9D or =E2=80=9CMonitored int= erfaces=E2=80=9D. Internet > > > > should be RED because it is called RED everywhere else. > > > ACK > > >=20 > > > > * We will need to grep for Snort everywhere and replace that > > > > phrase > > > > with =E2=80=9CIntrusion Prevention System=E2=80=9D. I do not think th= at we have > > > > to > > > > mention suricata here instead. It is confusing when the menu > > > > option > > > > is called IDS. > > > Most work on this already should be done, I'll remove the last > > > existing > > > snort entries if I'll found one.=20 > > Cool. > >=20 > > > > * The menu option should be called IPS instead of IDS I think > > > > since > > > > IPS mode is default. Spelled out obviously. > > > ACK - It should be a very easy task to switch the language string > > > from > > > "Intrusion Detection System" to the desired "Intrusion Prevention > > > System". > > Cool. I am just saying that it needs to be done :) don=E2=80=99t forget t= he > > logging section. > >=20 > > > > Apart from that I have to say that this is really really good > > > > work, > > > > Stefan. The main thing is: it works. I think this is a huge > > > > improvement over what we had in the past and that this is > > > > bringing > > > > IPFire to the next level. It solves many problems that we > > > > didn=E2=80=99t dare > > > > to touch because snort was simply badly integrated into IPFire > > > > and > > > > did not really do much instead of wasting CPU cycles and > > > > memory. > > > >=20 > > > > I hope that we can continue working in this direction and maybe > > > > add > > > > some nice reports and so on and obviously carry over this work > > > > into > > > > IPFire 3 as soon as possible. > > > >=20 > > > > Please feel free to disagree with me above, but please do that > > > > via > > > > email if possible. > > > >=20 > > > > I want to target merging this into next as soon as possible in > > > > January. I hope that we can get the things above sorted out > > > > over the > > > > holidays since it should not be much work. > > > >=20 > > > Best regards, > > >=20 > > > -Stefan > > > > > On 17 Dec 2018, at 19:08, Stefan Schantl < > > > > > stefan.schantl(a)ipfire.org > > > > > > wrote: > > > > > > What is the bypass mark for? At least that should be set to > > > > > > nothing > > > > > > instead of one, because that will again conflict with the > > > > > > SNAT > > > > > > fix > > > > > > rule. > > > > > >=20 > > > > > > Best, > > > > > > -Michael > > > > > >=20 > > > > > Hello Michael, > > > > >=20 > > > > > taken from the official suricata.yaml documentation: > > > > >=20 > > > > > bypass mark and mask can be used to implement NFQ bypass. If > > > > > bypass > > > > > mark is set then the NFQ bypass is activated. Suricata will > > > > > set the > > > > > bypass mark/mask on packet of a flow that need to be > > > > > bypassed. The > > > > > Nefilter ruleset has to directly accept all packets of a flow > > > > > once > > > > > a > > > > > packet has been marked. > > > > Weird. Why would you even pass this to the userspace process > > > > then? > > > >=20 > > > > Whatever, we don=E2=80=99t need it then... > > > >=20 > > > > > In my commit, I've disabled the bypass options to exactly > > > > > prevent > > > > > from > > > > > any issuse again with the SNAT packet mark. > > > > >=20 > > > > > Best regards, > > > > >=20 > > > > > -Stefan > > > > Best, > > > > -Michael > > > >=20 > > > > > > > On 17 Dec 2018, at 14:21, Stefan Schantl < > > > > > > > stefan.schantl(a)ipfire.org > > > > > > > > wrote: > > > > > > > Am Sonntag, den 16.12.2018, 21:28 +0100 schrieb Peter > > > > > > > M=C3=BCller: > > > > > > > > Hello Stefan, > > > > > > > >=20 > > > > > > > > to be a bit more precise about the NAT issue: > > > > > > > >=20 > > > > > > > > My setup is the IPFire Suricata test VM running in KVM, > > > > > > > > with > > > > > > > > two clients (Debian and OpenBSD) directly attached to > > > > > > > > it. > > > > > > > >=20 > > > > > > > > The Debian machine is located in GREEN, OpenBSD in > > > > > > > > ORANGE. > > > > > > > > RED interface is connected via bridge to my actual > > > > > > > > testing > > > > > > > > LAN; > > > > > > > > for the first testing, any outgoing traffic to the > > > > > > > > internet > > > > > > > > was allowed (I will test upstream proxy behaviour > > > > > > > > later). > > > > > > > >=20 > > > > > > > > While GREEN was using IPv4 range 192.168.100.0/24, with > > > > > > > > IPFire > > > > > > > > as 192.168.100.1, enabling Suricata caused packets > > > > > > > > coming > > > > > > > > from > > > > > > > > GREEN not to be NATted anymore: Instead of using the > > > > > > > > firewall's > > > > > > > > RED IP for destination, it was the internal GREEN IP. > > > > > > > >=20 > > > > > > > > Let me know whether is is useful or not. :-) > > > > > > > >=20 > > > > > > > > Thanks, and best regards, > > > > > > > > Peter M=C3=BCller > > > > > > > >=20 > > > > > > > > > Hello Stefan, > > > > > > > > >=20 > > > > > > > > > back again. :-) > > > > > > > > >=20 > > > > > > > > > The new IDS WebUI looks quite good so far - > > > > > > > > > enabling/disabling > > > > > > > > > Suricata works as well as selecting the rule source > > > > > > > > > and the > > > > > > > > > operation mode (IDS/IPS). > > > > > > > > >=20 > > > > > > > > > I was also able to download the "Snort/VRT Community" > > > > > > > > > ruleset. > > > > > > > > > Trying to switch to the "Emerging Threats" ruleset is > > > > > > > > > possible, > > > > > > > > > but downloading its ruleset afterwards is not: The > > > > > > > > > GUI > > > > > > > > > simply > > > > > > > > > stalls, printing a message that "Snort (!) is > > > > > > > > > performing a > > > > > > > > > task". > > > > > > > > >=20 > > > > > > > > > The WebUI services page still shows IDS status for > > > > > > > > > each > > > > > > > > > interface, > > > > > > > > > which does not seem to work anymore (everything is > > > > > > > > > stopped, > > > > > > > > > but > > > > > > > > > Suricata was active on RED and GREEN). > > > > > > > > >=20 > > > > > > > > > Further, a client located in GREEN behind the test > > > > > > > > > firewall > > > > > > > > > instance is unable to browse the internet as soon as > > > > > > > > > Suricata > > > > > > > > > is > > > > > > > > > enabled. If disabled, downloading ET rulesets work as > > > > > > > > > well > > > > > > > > > as > > > > > > > > > internet traffic. At the moment, I am flying blind > > > > > > > > > here, > > > > > > > > > but it > > > > > > > > > looks > > > > > > > > > like packets are not NATted anymore if Suricata is > > > > > > > > > active. > > > > > > > > >=20 > > > > > > > > > Any outgoing connection is in state "SYN_SENT" if > > > > > > > > > Suricata > > > > > > > > > is > > > > > > > > > active. > > > > > > > > >=20 > > > > > > > > > A portscan against the firewall (GREEN interface) is > > > > > > > > > not > > > > > > > > > detected, > > > > > > > > > even though ET SCAN ruleset is enabled (used nmap > > > > > > > > > with NSE > > > > > > > > > active). > > > > > > > > >=20 > > > > > > > > > Especially the outgoing connection/NAT/? issue > > > > > > > > > mentioned > > > > > > > > > above > > > > > > > > > breaks things in my scenario. Anything else are minor > > > > > > > > > issues > > > > > > > > > (of > > > > > > > > > course, a portscan should be detected, this needs > > > > > > > > > further > > > > > > > > > investigation > > > > > > > > > indeed). WebUI works fine so far. > > > > > > > > >=20 > > > > > > > > > Thanks again for your work; I hope the feedback can > > > > > > > > > appreciate > > > > > > > > > it > > > > > > > > > somehow. :-) > > > > > > > > >=20 > > > > > > > > > Let me know if there are questions. > > > > > > > > >=20 > > > > > > > > > Thanks, and best regards, > > > > > > > > > Peter M=C3=BCller > > > > > > > > >=20 > > > > > > > Hello Peter, > > > > > > >=20 > > > > > > > thanks for testing and your feedback. > > > > > > >=20 > > > > > > > I've setup a test environment and was able to re-produce > > > > > > > your > > > > > > > NAT > > > > > > > issue > > > > > > > very easy. After some research and help of Michael, we > > > > > > > were > > > > > > > able to > > > > > > > find the real issue. > > > > > > >=20 > > > > > > > It was located in a identical mark on the packets which > > > > > > > already > > > > > > > have > > > > > > > been scanned by suricata and packets which should be > > > > > > > modified > > > > > > > for > > > > > > > SNAT > > > > > > > in the "Mangle" table of the routing logic by the kernel. > > > > > > >=20 > > > > > > > The changes can be found in my git repository: > > > > > > >=20 > > > > > > > https://git.ipfire.org/?p=3Dpeople/stevee/ipfire-2.x.git;a=3Dco= mmit;h=3Df5ad510e3c0f416a1507999f5ad20ab171df9c07 > > > > > > >=20 > > > > > > > I'll upload a fixed image very soon - please keep on > > > > > > > testing. > > > > > > >=20 > > > > > > > Best regards, > > > > > > >=20 > > > > > > > -Stefan >=20 >=20 --===============2690295716031859180== Content-Type: application/pgp-signature Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="signature.asc" MIME-Version: 1.0 LS0tLS1CRUdJTiBQR1AgU0lHTkFUVVJFLS0tLS0KCmlRSXpCQUFCQ2dBZEZpRUVXTzBOWHRTcnZo YXN5dERuVHRkT0ZZK1RzdDRGQWx3aWdrUUFDZ2tRVHRkT0ZZK1QKc3Q3SzZSQUFpeUxrREJ5elZP YzZKWTQrcko1RlV6OHBFL2lDQVZENkVFSXk2UzVSRUhRTWFxV0xQbFpxMDJqMwpCMzhpV2s4UnlF cU5lN3hIeml2Vi96a3R4S1B5MzdMc0lGOUV5WnJncDNEZ2lObXpsMmlsTFpuUXNPSEJrUVBhCllx WkpxQ1hRNlBOWVdYNXc4YzBYVTVnenZXWTY4VitKTWw4TytUSVY4VWV3eFJmcEE1cjNQRlMyRTg1 Zi81RmcKdXg0TU1WczdvRTRNQThadHZKY0d4eE1neXdLMVZRb3R2U29JU3MwQzY1YjFPdzI2OGtB NHZqZmVZdUx3QkQxagpuSExlUUkxU29kVkZPNkVpUURJSXZwQTdYcGx2eDM5Q0tRVkhQckgxUGVY WVFhU2tYQkhUQXVNSUJkUjQ0MGROCld3eXR1aTJDYzVmdDcvTm5RTlZXakRKSWhSZXZBWDNoQy9L WGhSMUI0bHNmWVQweFdzZWtJK3FwV21aWHBKVGIKQVhReWRUZjVpeEU3SFlBUEtqSCtZL0V5dE5v NkZCKyt2VXRHQjhlTHlqTkRPUTJrTmcwdmF1MWdmS3owQ0VuegpBamRPcEJkaUR1ek1aSXJvR3hM SDNhRFJNZGt3YnUvYzgyb0kzbnVxVUFJRERpMmdRQml6MjFjRHBRRTJONHZBCnNEOVM5dWd6RENo RHg3YzY3azlVWmxWOVBVYUc2azlvUGdiUjJQblRQMjlYS241NDEvSVVkVi83QTFRQVduZGoKOEZY OS83d2lJdnNJY2YxN2laTGVOUHY1QzBlbGlWZXVBRytZbHViYVZveDE4ZjRzdlFlZ09PV01LUmwy SUlCawowUHVSdER6TzU0L3owdTJIMGVyaldDR0x2anZ3UEx4ekx4L1JoVzEyRUVsZXphZXFSMTA9 Cj1tVWVxCi0tLS0tRU5EIFBHUCBTSUdOQVRVUkUtLS0tLQo= --===============2690295716031859180==--