Traditional Network Intrusion Detection Systems (NIDS), which became popular in the late 1990s, still have limited security efficacy. I will briefly discuss the issues limiting NIDS effectiveness, attempts at improvements that provided only minimal incremental advances, the underlying design flaw, and a new approach that shows a lot of promise.
The initial problem with signature-based NIDS was “tuning.” When they are tuned too loosely, they generate too many false positives. When tuned too tightly, false negatives become a problem. Furthermore, most organizations don’t have the resources for continual tuning.
In the early 2000s, Security Information and Event Management (SIEM) systems were developed, in part, to address this issue. The idea was to leave the NIDS loosely tuned, and let the SIEM’s analytics figure out what’s real and what’s not by correlating with other log sources and contextual information. Unfortunately, SIEMs still generate too many false positives, and can only alert on predefined patterns of attack.
Over the next 15+ years, there have been several innovations in NIDS and network security. These include (1) using operating system and application data to reduce false positives and reduce tuning efforts, (2) complementing NIDS with sandboxed file detonation, (3) adding machine learning based static file analysis, and (4) reducing the network attack surface with next generation firewalls.
There has also been a school of thought claiming anomaly detection is the answer to complement or even replace signature-based NIDS. Different statistical approaches have been tried over the years, the latest being various types of machine learning.
However, we are still seeing far too many successful cyber attacks that take weeks and even months to detect. The question is why?
The underlying design flaw for the last 20 years in virtually all NIDS is that they are restricted to examining an individual packet at line speed, deciding if it’s good or bad, and then going on to the next packet. There are some NIDS that are session oriented rather than stream oriented, but the decision-making time frame is still line speed. If malware or a protocol violation is detected, the NIDS generates an alert. Typically the alert is sent to a log repository or SIEM for further analysis.
This approach means that network security expert have been limiting themselves to detection algorithms that can be used in an appliance (physical or virtual) at line speeds. This is the key flaw that must be addressed in order to enable these experts to build an advanced NIDS with dramatically improved efficacy.
The cost effectiveness of cloud computing and storage makes a new kind of NIDS possible. Now all you need on the network, at the perimeter, on internal subnets, and/or in your cloud environment, are lightweight software sensors to capture, filter, and compress full packet streams. The captured packets are stored in the cloud. This means that traditional line speed analysis is only step one.
The full packets are available for retrospective analysis based on new incoming threat intelligence. And because full packets are available, threat intelligence not only includes IP addresses, URLs, and file hashes, but new signatures built to detect attacks on newly discovered vulnerabilities. Ideally, this is done automatically by the vendor with no effort required by the customer. Also, adding new signatures from your own sources is supported.
Furthermore, additional methods of analysis are performed including anomaly detection, machine learning static file analysis, and sandboxed file detonation. And the solution uses multiple correlation methods to analyze the results of these different processes. In other words, initially detected weak signals are correlated to generate strong signals with a low false positive rate and without increasing false negatives.
Alerts are categorized using the Lockheed Martin Kill Chain™ to enable the SOC analyst to prioritize his efforts. The NIDS user interface provides layered levels of detail, down the packets if necessary, showing why the alert was generated, thus shortening the time needed to triage each alert.
Finally, new methods of analysis can be added in the cloud without worrying about on-premise appliances having to be forklift upgraded due to added processing and/or memory requirements.
This is the type of solution I recommend you evaluate to reduce the risk of successful cyber attacks. Dare I call it a “Next Generation NIDS?” “Advanced NIDS?” What would you call this solution?
This article was originally posted on LinkedIn. https://www.linkedin.com/pulse/evolution-network-intrusion-detection-bill-frank?trk=mp-author-card
