From mboxrd@z Thu Jan 1 00:00:00 1970 From: Tim FitzGeorge To: development@lists.ipfire.org Subject: Re: IPFire meets Suricata - Call for tester Date: Fri, 21 Dec 2018 16:03:29 +0000 Message-ID: <7e6e3633-58bf-cbcc-9c8c-918dbd9e0838@tfitzgeorge.me.uk> In-Reply-To: <3F5A20DB-12BB-4E78-A174-4740A7E9E6FD@ipfire.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1790028706185988435==" List-Id: --===============1790028706185988435== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi, I've loaded a system up with the test image.=C2=A0 I've not had much time to actually use it (hopefully after Christmas).=C2=A0 I would like to know when there'll be an image with the marking fixed. I agree with Michael over the GUI; splitting the system settings and ruleset settings seems like a good idea.=C2=A0 The thing that caught me originally was that I selected a ruleset and then clicked on 'Download new ruleset' and nothing happened because I hadn't clicked on 'Save'. A couple of points on automatic updates: 1 - The updates shouldn't run exactly on the hour since this means the load on the servers will be very bursty.=C2=A0 It would be better to spread the load by running at some point past the hour and ensuring that this is different for every system (generate a random offset during installation?).=C2=A0 Since the rulesets are sizeable (100 MiB for Talos VRT) and they're given away for free, I think it's a good idea to do something about this. 2 - There's a warning in the oinkmaster documentation: Oinkmaster can be run manually or automatically as a cron job. But of=20 course, be very careful when doing the latter. Things could easily get=20 messed up from one update to another because of different layout in the=20 rules archive, typo in a rule, bugs in the script, and so on. At least=20 run snort -T on the new rules before restarting your Snort process. Since this is about the rulefiles I'm sure it applies to Suricata as well.=C2=A0 I have actually seen one occasion when the updated rules failed snort -T with my updater - and was after the download passed an md5sum check.=C2=A0 So the downloaded ruleset can be corrupt, if not very often.=C2= =A0 Should probably do something like: download update backup old rules run oinkmaster if (suricata -T is OK) then =C2=A0 tell suricata to re-read rules else =C2=A0 restore backup end if Otherwise, I'm encouraged by the reduction in memory usage over snort, although it'll be interesting to see how this holds up with a larger ruleset and under heavier load. Regards, Tim On 20/12/2018 14:05, Michael Tremer wrote: > Hi, > >> On 20 Dec 2018, at 13:03, Stefan Schantl wro= te: >> >> Hello Michael, >> >> thanks for having a look at suricata and your feedback. >>> Hi, >>> >>> So I finally had a closer look at this, too. I have a couple of >>> questions and would like to propose some changes to the UI: >>> >>> Questions: >>> >>> * Is it a good idea to have =E2=80=9CNo=E2=80=9D rules as default and als= o =E2=80=9Cno=E2=80=9D >>> automatic rule updates? I guess we can configure at least Emerging >>> Threats as a good default because they are free. I also think that >>> when IDS is enabled, the system should update the ruleset by default >>> once a day. When IDS is enabled, there should be no rule update. >>> >>> * That also leads me to: Should no automatic updates even be an >>> option? >>> >> Making the free ruleset of emergingthreads as default is okay for me - >> ACK. >> >> I'm also fine with doing a weekly ruleset update as default setting. >> >> My personal opinion is, it should be possible to disable the automatic >> rule update. There are some situations where this feature could be >> quite usefull, for example if you have only volume limited internet >> access or even other (which we currently do not have in mind....). > I guess if you don=E2=80=99t update the rules there is no point in running = the IPS. It=E2=80=99s just like a virus scanner with outdated signatures: det= ecting even less. > > Do we have more people with an opinion on this? > > The rulesets are indeed quite large. But the only people who might want to = disable it are people who run IPFire on an IoT appliance and those are too sl= ow to run the IPS anyways. So it=E2=80=99s not a very strong argument in my o= pinion. > > The best argument I can make for it is only that this is probably unwanted = behaviour and therefore can easily be misconfugured. > >> >>> Bugs: >>> >>> * When I enabled IDS in IPS mode, I checked the box to activate it >>> and selected the ET ruleset and hit save. Suricata starts but without >>> rules which is definitely not what I wanted. Either the box should be >>> greyed out or the downloader is launched when there are no rules, >>> yet. >>> >>> * It is possible to enable suricata without selecting any interfaces. >>> That just wastes RAM and makes it easy to enable it without actually >>> having the protection. >>> >>> If you agree, I will create tickets for this. >> Thanks, fixed. >> >> https://git.ipfire.org/?p=3Dpeople/stevee/ipfire-2.x.git;a=3Dcommit;h=3Deb= dd0f9a90da800cc6173f6f30fb0621dddc354b > I am on a plane now, so I cannot see this, but thanks! > >>> UI Stuff: >>> >>> * I think that the settings to enable suricata up to and including >>> =E2=80=9CTraffic analysing=E2=80=9D should be in the top box. The second = box should >>> only care about the rules. That might even solve the problem >>> described above. >> During development I had a very similar idea, of grouping all suricata=20 >> settings and all which are for the ruleset together into seperate >> boxes. I dropped this idea, because of some big restrictions of the >> code and I was afraid of how many work this would be to re-write huge >> parts of it to get the desired result. > I think that this is very little work. > >>> * I find it quite confusing that one has to =E2=80=9CActivate Intrusion >>> Detection System=E2=80=9D and then again select IDS or IPS. It doesn=E2= =80=99t seem >>> to make much sense even to select the top box when IPS is checked. I >>> would therefore prefer the following: one box that just says >>> =E2=80=9CActivate=E2=80=9D and that switches on or off the whole thing, a= nd then a >>> second box that says =E2=80=9Cmonitor only=E2=80=9D or something along th= ese lines. >>> That box should be disabled by default which will run suricata in IPS >>> mode (which is what everybody wants!) and when checked will downgrade >>> it to IDS mode. Right now it looks like this is an equal level of >>> both options but I think that there is a risk here that people enable >>> IDS and think that they are protected when they are in fact not. I >>> think the =E2=80=9Cmonitor only=E2=80=9D checkbox makes this a lot cleare= r. >> Fair enough - I'm going now to rework the current section for choosing >> the runmode and move from the radio buttons to a select box which will >> be called "Only monitor traffic" as you suggested and will place it >> right next to the box where you can enable the whole IPS/IDS system. > I think the best wording is: Monitor traffic only > >>> * The Traffic Analyzing headline doesn=E2=80=99t make any sense. It should >>> rather be =E2=80=9CMonitored zones=E2=80=9D or =E2=80=9CMonitored interfa= ces=E2=80=9D. Internet >>> should be RED because it is called RED everywhere else. >> ACK >> >>> * We will need to grep for Snort everywhere and replace that phrase >>> with =E2=80=9CIntrusion Prevention System=E2=80=9D. I do not think that w= e have to >>> mention suricata here instead. It is confusing when the menu option >>> is called IDS. >> Most work on this already should be done, I'll remove the last existing >> snort entries if I'll found one.=20 > Cool. > >>> * The menu option should be called IPS instead of IDS I think since >>> IPS mode is default. Spelled out obviously. >> ACK - It should be a very easy task to switch the language string from >> "Intrusion Detection System" to the desired "Intrusion Prevention >> System". > Cool. I am just saying that it needs to be done :) don=E2=80=99t forget the= logging section. > >>> Apart from that I have to say that this is really really good work, >>> Stefan. The main thing is: it works. I think this is a huge >>> improvement over what we had in the past and that this is bringing >>> IPFire to the next level. It solves many problems that we didn=E2=80=99t = dare >>> to touch because snort was simply badly integrated into IPFire and >>> did not really do much instead of wasting CPU cycles and memory. >>> >>> I hope that we can continue working in this direction and maybe add >>> some nice reports and so on and obviously carry over this work into >>> IPFire 3 as soon as possible. >>> >>> Please feel free to disagree with me above, but please do that via >>> email if possible. >>> >>> I want to target merging this into next as soon as possible in >>> January. I hope that we can get the things above sorted out over the >>> holidays since it should not be much work. >>> >> Best regards, >> >> -Stefan >>>> On 17 Dec 2018, at 19:08, Stefan Schantl >>>> wrote: >>>>> What is the bypass mark for? At least that should be set to >>>>> nothing >>>>> instead of one, because that will again conflict with the SNAT >>>>> fix >>>>> rule. >>>>> >>>>> Best, >>>>> -Michael >>>>> >>>> Hello Michael, >>>> >>>> taken from the official suricata.yaml documentation: >>>> >>>> bypass mark and mask can be used to implement NFQ bypass. If bypass >>>> mark is set then the NFQ bypass is activated. Suricata will set the >>>> bypass mark/mask on packet of a flow that need to be bypassed. The >>>> Nefilter ruleset has to directly accept all packets of a flow once >>>> a >>>> packet has been marked. >>> Weird. Why would you even pass this to the userspace process then? >>> >>> Whatever, we don=E2=80=99t need it then... >>> >>>> In my commit, I've disabled the bypass options to exactly prevent >>>> from >>>> any issuse again with the SNAT packet mark. >>>> >>>> Best regards, >>>> >>>> -Stefan >>> Best, >>> -Michael >>> >>>>>> On 17 Dec 2018, at 14:21, Stefan Schantl < >>>>>> stefan.schantl(a)ipfire.org >>>>>>> wrote: >>>>>> Am Sonntag, den 16.12.2018, 21:28 +0100 schrieb Peter M=C3=BCller: >>>>>>> Hello Stefan, >>>>>>> >>>>>>> to be a bit more precise about the NAT issue: >>>>>>> >>>>>>> My setup is the IPFire Suricata test VM running in KVM, with >>>>>>> two clients (Debian and OpenBSD) directly attached to it. >>>>>>> >>>>>>> The Debian machine is located in GREEN, OpenBSD in ORANGE. >>>>>>> RED interface is connected via bridge to my actual testing >>>>>>> LAN; >>>>>>> for the first testing, any outgoing traffic to the internet >>>>>>> was allowed (I will test upstream proxy behaviour later). >>>>>>> >>>>>>> While GREEN was using IPv4 range 192.168.100.0/24, with >>>>>>> IPFire >>>>>>> as 192.168.100.1, enabling Suricata caused packets coming >>>>>>> from >>>>>>> GREEN not to be NATted anymore: Instead of using the >>>>>>> firewall's >>>>>>> RED IP for destination, it was the internal GREEN IP. >>>>>>> >>>>>>> Let me know whether is is useful or not. :-) >>>>>>> >>>>>>> Thanks, and best regards, >>>>>>> Peter M=C3=BCller >>>>>>> >>>>>>>> Hello Stefan, >>>>>>>> >>>>>>>> back again. :-) >>>>>>>> >>>>>>>> The new IDS WebUI looks quite good so far - >>>>>>>> enabling/disabling >>>>>>>> Suricata works as well as selecting the rule source and the >>>>>>>> operation mode (IDS/IPS). >>>>>>>> >>>>>>>> I was also able to download the "Snort/VRT Community" >>>>>>>> ruleset. >>>>>>>> Trying to switch to the "Emerging Threats" ruleset is >>>>>>>> possible, >>>>>>>> but downloading its ruleset afterwards is not: The GUI >>>>>>>> simply >>>>>>>> stalls, printing a message that "Snort (!) is performing a >>>>>>>> task". >>>>>>>> >>>>>>>> The WebUI services page still shows IDS status for each >>>>>>>> interface, >>>>>>>> which does not seem to work anymore (everything is stopped, >>>>>>>> but >>>>>>>> Suricata was active on RED and GREEN). >>>>>>>> >>>>>>>> Further, a client located in GREEN behind the test firewall >>>>>>>> instance is unable to browse the internet as soon as >>>>>>>> Suricata >>>>>>>> is >>>>>>>> enabled. If disabled, downloading ET rulesets work as well >>>>>>>> as >>>>>>>> internet traffic. At the moment, I am flying blind here, >>>>>>>> but it >>>>>>>> looks >>>>>>>> like packets are not NATted anymore if Suricata is active. >>>>>>>> >>>>>>>> Any outgoing connection is in state "SYN_SENT" if Suricata >>>>>>>> is >>>>>>>> active. >>>>>>>> >>>>>>>> A portscan against the firewall (GREEN interface) is not >>>>>>>> detected, >>>>>>>> even though ET SCAN ruleset is enabled (used nmap with NSE >>>>>>>> active). >>>>>>>> >>>>>>>> Especially the outgoing connection/NAT/? issue mentioned >>>>>>>> above >>>>>>>> breaks things in my scenario. Anything else are minor >>>>>>>> issues >>>>>>>> (of >>>>>>>> course, a portscan should be detected, this needs further >>>>>>>> investigation >>>>>>>> indeed). WebUI works fine so far. >>>>>>>> >>>>>>>> Thanks again for your work; I hope the feedback can >>>>>>>> appreciate >>>>>>>> it >>>>>>>> somehow. :-) >>>>>>>> >>>>>>>> Let me know if there are questions. >>>>>>>> >>>>>>>> Thanks, and best regards, >>>>>>>> Peter M=C3=BCller >>>>>>>> >>>>>> Hello Peter, >>>>>> >>>>>> thanks for testing and your feedback. >>>>>> >>>>>> I've setup a test environment and was able to re-produce your >>>>>> NAT >>>>>> issue >>>>>> very easy. After some research and help of Michael, we were >>>>>> able to >>>>>> find the real issue. >>>>>> >>>>>> It was located in a identical mark on the packets which already >>>>>> have >>>>>> been scanned by suricata and packets which should be modified >>>>>> for >>>>>> SNAT >>>>>> in the "Mangle" table of the routing logic by the kernel. >>>>>> >>>>>> The changes can be found in my git repository: >>>>>> >>>>>> https://git.ipfire.org/?p=3Dpeople/stevee/ipfire-2.x.git;a=3Dcommit;h= =3Df5ad510e3c0f416a1507999f5ad20ab171df9c07 >>>>>> >>>>>> I'll upload a fixed image very soon - please keep on testing. >>>>>> >>>>>> Best regards, >>>>>> >>>>>> -Stefan > --===============1790028706185988435==--