Hello, Good job. Stefan, are you planning to release another image very soon? -Michael > On 25 Dec 2018, at 20:17, Stefan Schantl wrote: > > Hello Tim, > > thanks for joining and providing your feedback on this list. >> Hi, >> >> I've loaded a system up with the test image. I've not had much time >> to >> actually use it (hopefully after Christmas). I would like to know >> when >> there'll be an image with the marking fixed. > > I'll generate and upload a new one very soon. > >> >> I agree with Michael over the GUI; splitting the system settings and >> ruleset settings seems like a good idea. The thing that caught me >> originally was that I selected a ruleset and then clicked on >> 'Download >> new ruleset' and nothing happened because I hadn't clicked on 'Save'. > > I splitted the section into two parts. One for configuring the IDS and > one which contains the rules settings for now. > >> >> A couple of points on automatic updates: >> >> 1 - The updates shouldn't run exactly on the hour since this means >> the >> load on the servers will be very bursty. It would be better to >> spread >> the load by running at some point past the hour and ensuring that >> this >> is different for every system (generate a random offset during >> installation?). Since the rulesets are sizeable (100 MiB for Talos >> VRT) >> and they're given away for free, I think it's a good idea to do >> something about this. > > Thanks for that, I don't know if "fcron" does any kind of > randomization. I'll have a closer look on it. > > I'm reading you are using the ruleset provided by the Talos team. I'm > intrested in adding support for this. Do you have some more details > about this ruleset and a download URL? > >> >> 2 - There's a warning in the oinkmaster documentation: >> >> Oinkmaster can be run manually or automatically as a cron job. But >> of >> course, be very careful when doing the latter. Things could easily >> get >> messed up from one update to another because of different layout in >> the >> rules archive, typo in a rule, bugs in the script, and so on. At >> least >> run snort -T on the new rules before restarting your Snort process. >> >> Since this is about the rulefiles I'm sure it applies to Suricata as >> well. I have actually seen one occasion when the updated rules >> failed >> snort -T with my updater - and was after the download passed an >> md5sum >> check. So the downloaded ruleset can be corrupt, if not very often. >> Should probably do something like: >> >> download update >> backup old rules >> run oinkmaster >> if (suricata -T is OK) then >> tell suricata to re-read rules >> else >> restore backup >> end if > > Thanks for the hint. I have to dig into this, if suricata offers a > mechanism to check the ruleset before doing a reload of it. > >> >> Otherwise, I'm encouraged by the reduction in memory usage over >> snort, >> although it'll be interesting to see how this holds up with a larger >> ruleset and under heavier load. > > Best regards, > > -Stefan >> >> Regards, >> Tim >> >> On 20/12/2018 14:05, Michael Tremer wrote: >>> Hi, >>> >>>> On 20 Dec 2018, at 13:03, Stefan Schantl < >>>> stefan.schantl(a)ipfire.org> wrote: >>>> >>>> Hello Michael, >>>> >>>> thanks for having a look at suricata and your feedback. >>>>> Hi, >>>>> >>>>> So I finally had a closer look at this, too. I have a couple of >>>>> questions and would like to propose some changes to the UI: >>>>> >>>>> Questions: >>>>> >>>>> * Is it a good idea to have “No” rules as default and also “no” >>>>> automatic rule updates? I guess we can configure at least >>>>> Emerging >>>>> Threats as a good default because they are free. I also think >>>>> that >>>>> when IDS is enabled, the system should update the ruleset by >>>>> default >>>>> once a day. When IDS is enabled, there should be no rule >>>>> update. >>>>> >>>>> * That also leads me to: Should no automatic updates even be an >>>>> option? >>>>> >>>> Making the free ruleset of emergingthreads as default is okay for >>>> me - >>>> ACK. >>>> >>>> I'm also fine with doing a weekly ruleset update as default >>>> setting. >>>> >>>> My personal opinion is, it should be possible to disable the >>>> automatic >>>> rule update. There are some situations where this feature could >>>> be >>>> quite usefull, for example if you have only volume limited >>>> internet >>>> access or even other (which we currently do not have in >>>> mind....). >>> I guess if you don’t update the rules there is no point in running >>> the IPS. It’s just like a virus scanner with outdated signatures: >>> detecting even less. >>> >>> Do we have more people with an opinion on this? >>> >>> The rulesets are indeed quite large. But the only people who might >>> want to disable it are people who run IPFire on an IoT appliance >>> and those are too slow to run the IPS anyways. So it’s not a very >>> strong argument in my opinion. >>> >>> The best argument I can make for it is only that this is probably >>> unwanted behaviour and therefore can easily be misconfugured. >>> >>>>> Bugs: >>>>> >>>>> * When I enabled IDS in IPS mode, I checked the box to activate >>>>> it >>>>> and selected the ET ruleset and hit save. Suricata starts but >>>>> without >>>>> rules which is definitely not what I wanted. Either the box >>>>> should be >>>>> greyed out or the downloader is launched when there are no >>>>> rules, >>>>> yet. >>>>> >>>>> * It is possible to enable suricata without selecting any >>>>> interfaces. >>>>> That just wastes RAM and makes it easy to enable it without >>>>> actually >>>>> having the protection. >>>>> >>>>> If you agree, I will create tickets for this. >>>> Thanks, fixed. >>>> >>>> https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=ebdd0f9a90da800cc6173f6f30fb0621dddc354b >>> I am on a plane now, so I cannot see this, but thanks! >>> >>>>> UI Stuff: >>>>> >>>>> * I think that the settings to enable suricata up to and >>>>> including >>>>> “Traffic analysing” should be in the top box. The second box >>>>> should >>>>> only care about the rules. That might even solve the problem >>>>> described above. >>>> During development I had a very similar idea, of grouping all >>>> suricata >>>> settings and all which are for the ruleset together into seperate >>>> boxes. I dropped this idea, because of some big restrictions of >>>> the >>>> code and I was afraid of how many work this would be to re-write >>>> huge >>>> parts of it to get the desired result. >>> I think that this is very little work. >>> >>>>> * I find it quite confusing that one has to “Activate Intrusion >>>>> Detection System” and then again select IDS or IPS. It doesn’t >>>>> seem >>>>> to make much sense even to select the top box when IPS is >>>>> checked. I >>>>> would therefore prefer the following: one box that just says >>>>> “Activate” and that switches on or off the whole thing, and >>>>> then a >>>>> second box that says “monitor only” or something along these >>>>> lines. >>>>> That box should be disabled by default which will run suricata >>>>> in IPS >>>>> mode (which is what everybody wants!) and when checked will >>>>> downgrade >>>>> it to IDS mode. Right now it looks like this is an equal level >>>>> of >>>>> both options but I think that there is a risk here that people >>>>> enable >>>>> IDS and think that they are protected when they are in fact >>>>> not. I >>>>> think the “monitor only” checkbox makes this a lot clearer. >>>> Fair enough - I'm going now to rework the current section for >>>> choosing >>>> the runmode and move from the radio buttons to a select box which >>>> will >>>> be called "Only monitor traffic" as you suggested and will place >>>> it >>>> right next to the box where you can enable the whole IPS/IDS >>>> system. >>> I think the best wording is: Monitor traffic only >>> >>>>> * The Traffic Analyzing headline doesn’t make any sense. It >>>>> should >>>>> rather be “Monitored zones” or “Monitored interfaces”. Internet >>>>> should be RED because it is called RED everywhere else. >>>> ACK >>>> >>>>> * We will need to grep for Snort everywhere and replace that >>>>> phrase >>>>> with “Intrusion Prevention System”. I do not think that we have >>>>> to >>>>> mention suricata here instead. It is confusing when the menu >>>>> option >>>>> is called IDS. >>>> Most work on this already should be done, I'll remove the last >>>> existing >>>> snort entries if I'll found one. >>> Cool. >>> >>>>> * The menu option should be called IPS instead of IDS I think >>>>> since >>>>> IPS mode is default. Spelled out obviously. >>>> ACK - It should be a very easy task to switch the language string >>>> from >>>> "Intrusion Detection System" to the desired "Intrusion Prevention >>>> System". >>> Cool. I am just saying that it needs to be done :) don’t forget the >>> logging section. >>> >>>>> Apart from that I have to say that this is really really good >>>>> work, >>>>> Stefan. The main thing is: it works. I think this is a huge >>>>> improvement over what we had in the past and that this is >>>>> bringing >>>>> IPFire to the next level. It solves many problems that we >>>>> didn’t dare >>>>> to touch because snort was simply badly integrated into IPFire >>>>> and >>>>> did not really do much instead of wasting CPU cycles and >>>>> memory. >>>>> >>>>> I hope that we can continue working in this direction and maybe >>>>> add >>>>> some nice reports and so on and obviously carry over this work >>>>> into >>>>> IPFire 3 as soon as possible. >>>>> >>>>> Please feel free to disagree with me above, but please do that >>>>> via >>>>> email if possible. >>>>> >>>>> I want to target merging this into next as soon as possible in >>>>> January. I hope that we can get the things above sorted out >>>>> over the >>>>> holidays since it should not be much work. >>>>> >>>> Best regards, >>>> >>>> -Stefan >>>>>> On 17 Dec 2018, at 19:08, Stefan Schantl < >>>>>> stefan.schantl(a)ipfire.org >>>>>>> wrote: >>>>>>> What is the bypass mark for? At least that should be set to >>>>>>> nothing >>>>>>> instead of one, because that will again conflict with the >>>>>>> SNAT >>>>>>> fix >>>>>>> rule. >>>>>>> >>>>>>> Best, >>>>>>> -Michael >>>>>>> >>>>>> Hello Michael, >>>>>> >>>>>> taken from the official suricata.yaml documentation: >>>>>> >>>>>> bypass mark and mask can be used to implement NFQ bypass. If >>>>>> bypass >>>>>> mark is set then the NFQ bypass is activated. Suricata will >>>>>> set the >>>>>> bypass mark/mask on packet of a flow that need to be >>>>>> bypassed. The >>>>>> Nefilter ruleset has to directly accept all packets of a flow >>>>>> once >>>>>> a >>>>>> packet has been marked. >>>>> Weird. Why would you even pass this to the userspace process >>>>> then? >>>>> >>>>> Whatever, we don’t need it then... >>>>> >>>>>> In my commit, I've disabled the bypass options to exactly >>>>>> prevent >>>>>> from >>>>>> any issuse again with the SNAT packet mark. >>>>>> >>>>>> Best regards, >>>>>> >>>>>> -Stefan >>>>> Best, >>>>> -Michael >>>>> >>>>>>>> On 17 Dec 2018, at 14:21, Stefan Schantl < >>>>>>>> stefan.schantl(a)ipfire.org >>>>>>>>> wrote: >>>>>>>> Am Sonntag, den 16.12.2018, 21:28 +0100 schrieb Peter >>>>>>>> Müller: >>>>>>>>> Hello Stefan, >>>>>>>>> >>>>>>>>> to be a bit more precise about the NAT issue: >>>>>>>>> >>>>>>>>> My setup is the IPFire Suricata test VM running in KVM, >>>>>>>>> with >>>>>>>>> two clients (Debian and OpenBSD) directly attached to >>>>>>>>> it. >>>>>>>>> >>>>>>>>> The Debian machine is located in GREEN, OpenBSD in >>>>>>>>> ORANGE. >>>>>>>>> RED interface is connected via bridge to my actual >>>>>>>>> testing >>>>>>>>> LAN; >>>>>>>>> for the first testing, any outgoing traffic to the >>>>>>>>> internet >>>>>>>>> was allowed (I will test upstream proxy behaviour >>>>>>>>> later). >>>>>>>>> >>>>>>>>> While GREEN was using IPv4 range 192.168.100.0/24, with >>>>>>>>> IPFire >>>>>>>>> as 192.168.100.1, enabling Suricata caused packets >>>>>>>>> coming >>>>>>>>> from >>>>>>>>> GREEN not to be NATted anymore: Instead of using the >>>>>>>>> firewall's >>>>>>>>> RED IP for destination, it was the internal GREEN IP. >>>>>>>>> >>>>>>>>> Let me know whether is is useful or not. :-) >>>>>>>>> >>>>>>>>> Thanks, and best regards, >>>>>>>>> Peter Müller >>>>>>>>> >>>>>>>>>> Hello Stefan, >>>>>>>>>> >>>>>>>>>> back again. :-) >>>>>>>>>> >>>>>>>>>> The new IDS WebUI looks quite good so far - >>>>>>>>>> enabling/disabling >>>>>>>>>> Suricata works as well as selecting the rule source >>>>>>>>>> and the >>>>>>>>>> operation mode (IDS/IPS). >>>>>>>>>> >>>>>>>>>> I was also able to download the "Snort/VRT Community" >>>>>>>>>> ruleset. >>>>>>>>>> Trying to switch to the "Emerging Threats" ruleset is >>>>>>>>>> possible, >>>>>>>>>> but downloading its ruleset afterwards is not: The >>>>>>>>>> GUI >>>>>>>>>> simply >>>>>>>>>> stalls, printing a message that "Snort (!) is >>>>>>>>>> performing a >>>>>>>>>> task". >>>>>>>>>> >>>>>>>>>> The WebUI services page still shows IDS status for >>>>>>>>>> each >>>>>>>>>> interface, >>>>>>>>>> which does not seem to work anymore (everything is >>>>>>>>>> stopped, >>>>>>>>>> but >>>>>>>>>> Suricata was active on RED and GREEN). >>>>>>>>>> >>>>>>>>>> Further, a client located in GREEN behind the test >>>>>>>>>> firewall >>>>>>>>>> instance is unable to browse the internet as soon as >>>>>>>>>> Suricata >>>>>>>>>> is >>>>>>>>>> enabled. If disabled, downloading ET rulesets work as >>>>>>>>>> well >>>>>>>>>> as >>>>>>>>>> internet traffic. At the moment, I am flying blind >>>>>>>>>> here, >>>>>>>>>> but it >>>>>>>>>> looks >>>>>>>>>> like packets are not NATted anymore if Suricata is >>>>>>>>>> active. >>>>>>>>>> >>>>>>>>>> Any outgoing connection is in state "SYN_SENT" if >>>>>>>>>> Suricata >>>>>>>>>> is >>>>>>>>>> active. >>>>>>>>>> >>>>>>>>>> A portscan against the firewall (GREEN interface) is >>>>>>>>>> not >>>>>>>>>> detected, >>>>>>>>>> even though ET SCAN ruleset is enabled (used nmap >>>>>>>>>> with NSE >>>>>>>>>> active). >>>>>>>>>> >>>>>>>>>> Especially the outgoing connection/NAT/? issue >>>>>>>>>> mentioned >>>>>>>>>> above >>>>>>>>>> breaks things in my scenario. Anything else are minor >>>>>>>>>> issues >>>>>>>>>> (of >>>>>>>>>> course, a portscan should be detected, this needs >>>>>>>>>> further >>>>>>>>>> investigation >>>>>>>>>> indeed). WebUI works fine so far. >>>>>>>>>> >>>>>>>>>> Thanks again for your work; I hope the feedback can >>>>>>>>>> appreciate >>>>>>>>>> it >>>>>>>>>> somehow. :-) >>>>>>>>>> >>>>>>>>>> Let me know if there are questions. >>>>>>>>>> >>>>>>>>>> Thanks, and best regards, >>>>>>>>>> Peter Müller >>>>>>>>>> >>>>>>>> Hello Peter, >>>>>>>> >>>>>>>> thanks for testing and your feedback. >>>>>>>> >>>>>>>> I've setup a test environment and was able to re-produce >>>>>>>> your >>>>>>>> NAT >>>>>>>> issue >>>>>>>> very easy. After some research and help of Michael, we >>>>>>>> were >>>>>>>> able to >>>>>>>> find the real issue. >>>>>>>> >>>>>>>> It was located in a identical mark on the packets which >>>>>>>> already >>>>>>>> have >>>>>>>> been scanned by suricata and packets which should be >>>>>>>> modified >>>>>>>> for >>>>>>>> SNAT >>>>>>>> in the "Mangle" table of the routing logic by the kernel. >>>>>>>> >>>>>>>> The changes can be found in my git repository: >>>>>>>> >>>>>>>> https://git.ipfire.org/?p=people/stevee/ipfire-2.x.git;a=commit;h=f5ad510e3c0f416a1507999f5ad20ab171df9c07 >>>>>>>> >>>>>>>> I'll upload a fixed image very soon - please keep on >>>>>>>> testing. >>>>>>>> >>>>>>>> Best regards, >>>>>>>> >>>>>>>> -Stefan