Security Laboratory

Security Laboratory

Sound Practice in Intrusion Detection & Prevention using NitroSecurity

Michael Leland and Eric Knapp

Sound Practice in

Intrusion Detection & Prevention using NitroSecurity

Deployment Considerations

The first phase of any IPS deployment should be careful planning, including consideration of the network topology and expected traffic patterns as they might relate to risk. Answering these questions prior to IPS selection will help to ensure that an appropriate IPS solution is chosen. Factors to consider include:

  • What are the primary ingress/egress points of the network(s)? Typically, there is a single connection between the enterprise and the Internet, although in larger companies, there may be many, including dedicated connections to remote offices, and one or more connections to the Internet from any or all of these locations.
  • What critical assets are maintained on the network? Typically, the most critical assets are the information assets themselves: that is, the employee records, customer data, financial data, etc. In terms of regulatory compliance, employee information and financial data must be protected, with legal and/or financial consequences for those who do not comply. Other important assets include well-known services that may be susceptible to exploit—for example, DNS servers.
  • Where are they located physically and logically within the network? Knowledge of your network topology, and where important assets are located, is important for determining IPS device placement. An Intrusion Protection Device must be placed in-line to provide active blocking of malicious or unwanted traffic. Within a physical network (LAN) or logical networks (VLAN), there may be convenient aggregation points, where a multi-port IPS may be used to protect multiple areas & assets.
  • What is the physical connectivity/link speed of the network in key areas, and what is the expected traffic volumes in key areas? The physical layer is a simple consideration, yet it ultimately determines the interface types that must be supported on the chosen IPS. However, do not confuse throughput and link speed: for economical deployments, measure the average and peak traffic on a network link and use an appropriately sized IPS.
  • What protocols are used? Intrusion detection and prevention requires deep packet inspection, and therefore the average size and rate of packets is important. High-volume traffic with low packet sizes, such as VoIP, may put more strain on an IPS than streaming UDP video, which typically uses larger packet sizes.
  • What is the expected alert rate? Depending upon the network, there may anywhere from a few dozen to hundreds of thousands of events generated every hour. The amount of events generated will help determine both IPS placement, IPS device sizing, and signature tuning.
  • Are there business or operational associations to specific areas of the network? Different organizations use different applications, and use their networks in different ways. This means that an IPS policy (the set of active rules on the IPS) should differ between an administrative office and an engineering lab, for example. Another example: traffic within a dormitory might consist of allowed peer-to-peer traffic, gaming, etc., while traffic between a dorm room and other external networks might be prohibited. h Also, because your IPS is going to be generating events, it may be desirable to deploy separate IPS devices in relation to office locations, departments, or floor number, in order to simplify event management. This is especially useful when managing multiple IPS devices within a large network through a common user interface. For example, in a university, deploying an IPS in each dormitory (or specific wings or floors of dorms in larger units) will make it simple to track specific events back to a physical location, monitor specific student areas, etc.

Where to deploy an IPS

Intrusion Prevention Systems work in-line, and so must be installed on one (or more) network links that needs protection. Typically, the point of entry to the network, either directly upstream or directly downstream from the firewall, is a good choice for IPS. Critical assets, such as application server farms, are also good candidates for IPS protection. Finally, network services such as DNS, email, and groupware are good candidates for IPS protection.

Luckily, Virtual IPS operation allows a single IPS to protect different physical and/or logical connections from a single device. Virtual IPS creates separate virtual instances of the IPS engine, providing individual policies and actions to each. While it may still be a good idea to install multiple IPS devices, the use of Virtual IPSs can help to deploy better protection to more areas of the network.

Signature Tuning

IPS works as follows: packets are inspected and matched against complex rules, or signatures, to detect known, malicious patterns. Upon detection, the IPS can either alert (create an event for notification and later analysis), block (drop the packet, and create an event for notification and later analysis), or reset the connection (block the entire session, and create an event for notification and later analysis). The number of signatures, and the quality of those signatures, will directly affect how many events and notifications are generated, and how much traffic is blocked. Obviously, some traffic that may be appropriate for one group, might be considered inappropriate for others. For example, peer-to-peer traffic might be prohibited in most corporate networks, yet may be used heavily for collaboration and communication within and between engineering labs. Tuning your signature sets refers to the adjustment of policies to reduce false-positives while adequately protecting the network.

The first step to signature tuning is creating an accurate baseline of event activity on your network. The best method for this is to enable all rules, with the action set to “alert” only (not blocking or reset, as these actions may interrupt legitimate traffic using an un-tuned signature). After a period of time, the events can be analyzed, with the results being used to tune the signatures, either: editing the signature to make it more or less specific in its detection criteria; removing the rule from the active signature set; or leaving the signature as is. Once there is an understanding of expected event activity, rules can be set to block or reset (although in many cases, alert activity will be sufficient, such as when a signature indicates traffic that is of concern, but not inherently dangerous or prohibited).

Establishing baselines for anomaly detection

For flow-based signatures, understanding expected network traffic behavior is required. Luckily, NitroView is able to recommend thresholds based on observed network behavior. Operators of the NitroView application can designate a period of time representative of ‘normal’ network activity against which the anomaly wizard will be run. For some this might be a week while others may choose to process all flow transactions for a fiscal quarter (enterprise) or a given semester (education). The platform will evaluate all session statistics collected for this time range and return the baseline calculations for the following characteristics:

Inbound Connection Rate / Burst
Inbound Byte Rate
Inbound Packet Rate
Outbound Connection Rate / Burst
Outbound Byte Rate
Outbound Packet Rate
Connection Duration

By establishing the baseline for these values, rules can be implemented to perform such enforcement as rate limiting or bandwidth throttling as well as to identify to the networking or security professional when these specific thresholds have been exceeded. Default actions can be taken at the identification of flow activities exceeding the prescribed thresholds so that user traffic in violation of the ceiling can be contained and alerts to the effect can be addressed by the appropriate department.

Creating and deploying enterprise policies

Policies refer to sets of signatures used by the Intrusion Detection/Prevention System to determine threat and/or policy violations. Policies are useful when managing multiple IPSs, or when an IPS is protecting multiple network connections and a unique set of signatures is desired for each. In environments where a single IPS is used to analyze traffic on a single aggregate data path (such as between a perimeter firewall and the enterprise) there is a strong likelihood that the various user communities sharing the data path may have varying levels of policy enforcement necessary depending upon their business function and security requirements. It has been the obstacle of signature-based IDS/IPS solutions to effectively manage the deployment and maintenance of rule libraries where the necessary set of behaviors described by any one policy may contain thousands of unique signature values, each requiring an appropriate level of enforcement for each user community served by the device. The NitroGuard IPS has over 5,000 independent signatures, each requiring the operator to define an appropriate action and severity value for each of ten discreet attributes. This equates to over 50,000 policy decisions required by the security professional to determine an effective security posture in each IPS. Imagine a distributed enterprise with tens or even hundreds of sensors and the challenge becomes insurmountable without an effective policy management solution.

IPS Actions and their Impact

Prior to the introduction of Snort_inline in 2002, which was itself based upon the earlier Hogwash project, network based monitoring was limited to inspection of copied packets (i.e., on a span or mirror port). Known as Intrusion Detection Systems, or IDS, these systems would inspect packets, match them against known patterns (also referred to as rules or signatures), and provide an alert. With the introduction of inline detection, where the actual packets were being monitored (rather than a copy), the ability to take immediate action was provided. Intrusion Detection became Intrusion Prevention, because packets could now be dropped to prevent an attack from occurring. Subsequent advances introduced new prevention mechanisms, including TCP session resets, and most recently blacklisting. Today, the four primary actions that can be taken by an IPS are: Alert; Drop; Reset; and Blacklist.

Pass: The packet is ignored and allowed to pass through the IPS without any further action.

Alert: The suspect pack is allowed through the IPS without modification, and a notification is generated, typically via email, pager, or other means. The alert consists of a vendor-proprietary log format, for further investigation by a SEM (Security Event Management) or SIM (Security Information Management) system.

Activate: An alert rule that uses an extra condition to activate another dynamic rule when it is triggered.

Dynamic: A logging rule that is triggered by an activate rule. Together, activate and dynamic rules create an alert, and log additional session information that may be useful for post-analysis.

Block: The suspect packet is dropped.

Reset: The suspect packet is dropped and a TCP session reset tears down the entire session responsible for sending the offending packet.

Blacklist: All traffic from the source of the suspect packet (typically defined by Source IP or Source Port) is blocked for a defined period of time.

NOTE: Blacklisting utilizes the NitroGuard IPS's firewall to block traffic, and this operation differs from a normal IPS Block condition in three ways: it's ability to block subsequent traffic explicitly by source IP address and/or source port; it's ability to enforce the block on subsequent traffic for a period of time; and it's ability to block traffic conditionally. The final point is important, especially when used with rate-based or threshold-based signatures. Example: a signature to detect occurrences of 5 sequential DNS responses from a common source IP were set to block, all DNS responses would be blocked, and an alert would be generated after every fifth response. For this reason most threshold-signatures are set to alert only: in this example, setting the rule to 'block' would stop operation of all DNS services. However, using blacklisting, the signature can be set to 'alert', and the IPS will be able to then block subsequent traffic from that source after the condition has been met. This becomes important when attempting to prevent more complex attacks, and is discussed more in the following section, “pushing the boundaries of IPS.”

Pushing the boundaries of IPS

Advanced IPS capabilities

Virtual IPS or "multi Snort"

NitroGuard IPS supports virtual IPS operation, meaning that multiple instances of the IPS engine operate in a virtual code space. This theory is inline with that used by Google's Chrome browser, where each web browser window runs a complete set of processes—and for the same reason: running multiple distinct instances of a detection and prevention engine provide deployment, operational and performance benefits.

In terms of deployment, virtual IPS means that each interface (either a physical Ethernet interface or a VLAN) can support a complete set of configuration parameters, including which signatures are active, and what action is taken when a rule is triggered. This is a benefit when deploying NitroGuard IPS, as it allows a single appliance to protect multiple networks. Operationally, the distinct policies enforced by each virtual IPS can be tuned to suit the specific behavior patterns on, say, and engineering lab network versus a guest wireless network. Finally, in terms of performance, the use of separate IPS processes and parameters on each virtual IPS means that the operation of one virtual IPS will not affect another: that is, if (relatively) slow-firing threshold signatures are used extensively on one virtual IPS, they will not impact the performance of other signatures, because new engine processes can be used to maintain performance.

Flow collection

Flow collection is not an inherent function of an IPS. However, NitroSecurity believes that there is a need to understand the relationship between network activity (flows) and IPS events that might be seen on the managed network(s).

State Awareness

NitroGuard IPS includes a stateful firewall, useful for secondary defense against obfuscation and evasion. For example, in the example given below (Blocking the Kaminsky DNS Cache Poisoning Exploit), state awareness allows for protection against Denial of Service countermeasures.

Blacklisting – Internal Firewall Blocking

Blacklisting, described above, blocks traffic using a combination of signatures (to trigger a condition, based on one or more detected patterns) and a stateful firewall (to block session traffic relevant to the detected attack). This allows graceful application of a 'block' condition, only blocking relevant, suspect traffic after a condition is met, whereas normal Snort blocking would block all similar traffic. The distinction is very important when detecting attacks that use critical services, such as DNS (in fact, this is easiest to illustrate using DNS cache poisoning vulnerabilities, below).

Using IPS as a SIEM data-source

Many valuable data points may be generated using an IPS. Events themselves, at a minimum, should be directed to a SIEM for collection and analysis. Additional information may also be extrapolated, including suspect network addresses. These suspect talkers are determined through deep packet inspection of attacks such as DNS example, below. Suspect addresses may be compiled and used as the bases for further “alert” rules. For example, the IP address served from within an invalid DNS name resolution response is likely the IP address of malicious host: this host should be considered suspect, and any future communication with that host should issue a notification.

Examples of advanced Prevention Techniques

Blocking the Kaminsky DNS Cache Poisoning Exploit

The “Kaminsky” DNS cache poisoning exploit, well known from Dan Kaminsky's presentation at BlackHat and DefCon in August 2008, takes advantage of vulnerabilities in DNS – a trusted service that is ubiquitous throughout all networks. The exploit is relatively simple:

1. When a local DNS server does not know the IP of a requested name, it issues a query to an upstream server for resolution

2. The valid DNS name resolution request uses a unique transaction ID, which the requesting server uses to identify the paired response from the upstream server

3. Malicious responses are issued to similar domains, using,, These responses are valid DNS responses, although they contain a false resolution data, pointing the domain to a different IP address (typically owned by the hacker). These responses are acceptable to the requesting server, assuming the transaction ID matches. By using multiple subdomains, a flood of acceptable responses is generated, each with a random ID.

4. If a malicious response with the correct transaction ID wins the race condition, the requesting DNS server will cache the bogus result, directing subsequent domain requests to the false IP address.

5. When a local DNS server does not know the IP of a requested name, it issues a query to an upstream server for resolution

6. The valid DNS name resolution request uses a unique transaction ID, which the requesting server uses to identify the paired response from the upstream server

7. Malicious responses are issued to similar domains, using,, These responses are valid DNS responses, although they contain a false resolution data, pointing the domain to a different IP address (typically owned by the hacker). These responses are acceptable to the requesting server, assuming the transaction ID matches. By using multiple subdomains, a flood of acceptable responses is generated, each with a random ID.

8. If a malicious response with the correct transaction ID wins the race condition, the requesting DNS server will cache the bogus result, directing subsequent domain requests to the false IP address.

A simple “block” rule is insufficient, as the malicious DNS responses are valid: an IPS rule set to block these responses would prevent DNS from operating, causing most network services fail. Even threshold-based rules would block valid packets. However, by using threshold-based detection to detect the exploit, and using Blacklisting (which is technically a firewall feature within the NitroGuard IPS) to block the exploit, normal DNS operation is preserved.

More importantly, perhaps, is the importance of the evidence collection inherent in IPS alerting. While preventing exploitation is important, the offending DNS resolution responses contain valuable information: in this case, the IP address of a malicious host (the IP address that the exploiter is attempting to resolve your domain request to). While it can't be guaranteed that this IP is truly malicious (it could be a deliberate effort to direct traffic to another legitimate host, without knowledge or consent of that host), it is good policy to establish additional IPS detection rules, set to alert, whenever traffic is seen to or from that suspect IP. Luckily, an alert (with packet capture) allows to examine the DNS response that triggered the alert, an extract that IP address — which we can now use to put these new alerts in place.

Database Activity Monitoring

Database Activity Monitoring (often abbreviated as either DAM or DBM) refers to the detection and prevention of database transactions. Database Activity Monitors come in two forms: network- or agent- based. Agent-based DAM products reside on a database server, and supplement


Using the IPS as a data source among many

A SIEM is responsible for the overall examination of information and activity on a network, to spot anomalies and larger threat patterns, or “incidents.” Incidents may be thought of as “meta-events” – events that are built out of multiple smaller events, flows, and logs. A SIEM is only as valuable as the data that it manages, however. For this reason, IPS policies should establish robust notification when communicating with a SIEM. i.e., while you may not want to be personally notified of every event that occurs, it is important that as much data as possible be sent to the SIEM. The SIEM becomes responsible for filtering false positives, while retaining a rich knowledge-base from which to observe larger incidents.

Other data sources can compliment IPS events, providing context around a specific signature match. For example, host logs may contain valuable user name and authentication data. Network flows contain relevant information about where an event originated and where it is destined, and enables the analysis of root cause and the analysis of attack vectors: useful in tracking a virus outbreak or other malicious activity.

IPS event data and its use in correlation

One of the benefits of SIEM is the correlation of event data to determine larger, more complex attacks (and also to help eliminate false-positives). To do this, SIEMs use signatures similar to IPS signatures, where the patterns are made of other events rather than deep packet analysis. In other words, SIEM event correlation rules are like “meta events”, where the rule identifies a larger event (referred to as an “incident”) that is indicated by the presence of several other events in a known pattern. A very basic example is that of a brute force detection rule: the occurrence of several failed login attempts, followed by a successful login, all from the same source IP, could be indicative of a brute-force attack. By correlating many events (such as indivudal login failure notification) together, the data presented to the security team is much easier to decipher.

The role of IPS within overall Security Information & Event Management (SIEM) is important. Most IPS devices, such as the NitroGuard IPS, are able to alert on activity that can be useful within the correlation process. In addition, they are likely able to provide additional context—such as actual packet captures, and/or corresponding network flow data—that is useful in mitigating and remediating the incident, once it has been identified.

Using SIM to enhance IPS

Just as IPS device(s) benefit SIEM, SIEM can also benefit IPS operation. Because a SIEM will collect information from many devices—including multiple IPS devices, perhaps, as well as host logs, routers, application logs, firewalls, and even database session data—it is more capable of analyzing events holistically, and therefore capable of detecting more types of attacks than is possible via in-line packet inspection.

With remediation features—such as the dynamic application of new rules or blacklist conditions—the SIEM can therefore act as an additional detection layer, looking for broader threats over the entire infrastructure, and then applying point-defenses against those threats through an update to the IPS itself. This can be done automatically, although many consider it a better practice to remediate manually, in order to prevent the unintended blocking or disruption of potentially legitimate traffic.

Business applications

Regulatory drivers for IPS

Many compliance regulations, including the Payment Card Industry’s Digital Security Standard (“PCI-DSS”, often referred to as “PCI”), require adequate protection of confidential consumer information. In the case of PCI, the requirement for adequate network defenses (IPS, Firewall, etc.) are clearly defined, while in others the requirement for protection is left open to interpretation. In either case, documentation around IPS deployment, and the applicability of defined IPS policies to corporate security policies, is a valuable asset when facing a compliance audit.

IPS events and compliance reporting

Successful compliance to regulations such as PCI, Sarbanes-Oxley, HIPAA, and others is often reduced to successful reporting of activities as they pertain to confidential information and the access to that information. From this perspective, though an IPS is not explicitly defined as a requirement, the events generated from that IPS can be used to determine whether information systems have been compromised.