The SANS Technology Institute

Leadership Laboratory

The case for outsourcing Log Analysis

January 11th, 2008
By Stephen Northcutt

I recently saw a press release from SecureWorks that said: SecureWorks(R), a leading information security services provider protecting over 2,000 clients, has added a managed log retention service to a variety of its current managed, monitored and on- demand service offerings. The new service, driven by a partnership with an industry-leading log management solution provider LogLogic(R), will provide enterprises with comprehensive log aggregation, retention, searching and reporting.[1]

This is interesting: four or five years ago there were only a few log management vendors, LogLogic being one of them, and today there are a dozen, but I had not thought of this as a service. So, I went to Google, and the top hit on the day I tested was Counterpane. Here is their value proposition:

"As part of BT Counterpane's Managed Security Services, Log Management benefits customers in five key ways:

• Preserve 100% of logs in unaltered form, normalize security incidents and trends within BT Counterpane's Socrates correlation environment, and deliver immediate response.
• Satisfy explicit data retention requirements in many high-profile government and industry regulations.
• Enable alerting on huge volumes of raw log content without transmitting sensitive information outside the customer premises.
• A cost-effective solution to store and process terabytes of logs without investing in a costly SAN infrastructure.
• Provide a variety of pre-defined report templates, enabling our customers immediate utility without a time-consuming development cycle. "[2]

So, what is interesting about this? When I was first learning security, they told me there were four approaches to risk management - according to Shon Harris, the leading author for CISSP exam preparation:

"Senior management can then choose one of the following activities pertaining to each of the identified risks:

• Mitigate the risk by implementing the recommended countermeasure
• Accept the risk
• Avoid the risk
• Transfer the risk by purchasing insurance"[3]

So which of these four cases is managed log analysis? I would argue that it is an example of transfering the risk, only instead of purchasing insurance, you are purchasing a service. Back to the press release:

As part of the new partnership, SecureWorks will manage LogLogic appliances and integrate LogLogic functionality into the SecureWorks Client Portal for blended reporting and workflow capabilities. SecureWorks will also monitor and correlate the logs, collected by LogLogic appliances, to identify and respond to security threats. In addition, SecureWorks will resell LogLogic appliances and support contracts.

"In this current regulatory environment, the ability to digest all the data logs from various devices is essential for compliance purposes," said Corey Merchant, Vice President of Product Management at SecureWorks.

"LogLogic enhances our existing log monitoring by allowing us to collect, analyze and store all logs, not just security events of interest. It's just one more tool in our arsenal of managed security services we provide to keep clients safe and compliant. We are thrilled to be able to collaborate with LogLogic who has been paving the way for log management and intelligence for so long."

Note carefully the phrase above, "all logs"; this is a bit like a python desiring to eat a humpback whale. According to NIST SP 800-92, common sources of information include:

• "Antimalware Software. The most common form of antimalware software is antivirus software, which typically records all instances of detected malware, file and system disinfection attempts, and file quarantines.3 Additionally, antivirus software might also record when malware scans were performed and when antivirus signature or software updates occurred. Antispyware software and other types of antimalware software (e.g., rootkit detectors) are also common sources of security information.

• Intrusion Detection and Intrusion Prevention Systems. Intrusion detection and intrusion prevention systems record detailed information on suspicious behavior and detected attacks, as well as any actions intrusion prevention systems performed to stop malicious activity in progress. Some intrusion detection systems, such as file integrity checking software, run periodically instead of continuously, so they generate log entries in batches instead of on an ongoing basis.

• Remote Access Software. Remote access is often granted and secured through virtual private networking (VPN). VPN systems typically log successful and failed login attempts, as well as the dates and times each user connected and disconnected, and the amount of data sent and received in each user session. VPN systems that support granular access control, such as many Secure Sockets Layer (SSL) VPNs, may log detailed information about the use of resources.

• Web Proxies. Web proxies are intermediate hosts through which Web sites are accessed. Web proxies make Web page requests on behalf of users, and they cache copies of retrieved Web pages to make additional accesses to those pages more efficient. Web proxies can also be used to restrict Web access and to add a layer of protection between Web clients and Web servers. Web proxies often keep a record of all URLs accessed through them.

• Vulnerability Management Software. Vulnerability management software, which includes patch management software and vulnerability assessment software, typically logs the patch installation history and vulnerability status of each host, which includes known vulnerabilities and missing software updates.5 Vulnerability management software may also record additional information about hosts configurations. Vulnerability management software typically runs occasionally, not continuously, and is likely to generate large batches of log entries.

• Authentication Servers. Authentication servers, including directory servers and single sign-on servers, typically log each authentication attempt, including its origin, username, success or failure, and date and time.

• Routers. Routers may be configured to permit or block certain types of network traffic based on a policy. Routers that block traffic are usually configured to log only the most basic characteristics of blocked activity.

• Firewalls. Like routers, firewalls permit or block activity based on a policy; however, firewalls use much more sophisticated methods to examine network traffic. Firewalls can also track the state of network traffic and perform content inspection. Firewalls tend to have more complex policies and generate more detailed logs of activity than routers.

• Network Quarantine Servers. Some organizations check each remote host's security posture before allowing it to join the network. This is often done through a network quarantine server and agents placed on each host. Hosts that do not respond to the servers checks or that fail the checks are quarantined on a separate virtual local area network (VLAN) segment. Network quarantine servers log information about the status of checks, including which hosts were quarantined and for what reasons"[5]

Now just collecting and storing this much informmation is a real challenge, but what do you actually do with it? To find out, the SecurityLab approached LogLogic's Dr. Anton Chuvakin, "LogLogic can not only collect and cost-effectively store massive amounts of log data (up to 75,000 log messages a second per appliance), but also provides many ways of making sense of the data: reports that help ties logs to compliance mandates, index searches that allow unearthing specific pieces of log data during the investigation and real-time alerts that help highlight the critical log messages. What is more, open web API allows easy integration of LogLogic's platform with other application that need logs or reports based on logs."[7]

We can outsource the analysis and storage of logs, but we cannot outsource the responsibility or governance. According to NIST, bolding is mine, "Requirements and recommendations for logging should be created in conjunction with an analysis of the technology and resources needed to implement the log management process. Generally, organizations should require logging and analyzing the data that is of the greatest importance, and should also have non-mandatory recommendations for the other types and sources of data that should be logged and analyzed if time and resources permit. In some cases, organizations can choose to have all or nearly all of its log data generated and stored for at least a short period of time in case it is needed. This policy gives greater weight to security considerations than to usability and resource usage. Also this policy can support better decision making in some cases. When establishing requirements and recommendations, organizations should strive to be flexible since each system is different and will log different amounts of data than other systems within the organization."[8] These decisions cannot be outsourced!

Outsourcing has its challenges, so what is the benefit to an organization to outsource log analysis? Take just a second an look at the NIST list of common log sources above. Each of these information sources is different and most commonly they are written to syslog. Reading syslog entries is fairly cumbersome, writing syslog entry parsers is even harder, that is why there are so many companies selling products to do this. However, configuring syslog to collect information is fairly arcane, here is the core of a syslog configuration:

ifdef('LOGHOST', ,
user.err    /dev/console
user.err    /var/adm/messages
user.alert  'root, operator'
user.emerg  *
)[10]

So the driver to outsource log analysis is this, auditors who check for regulatory compliance are increasingly demanding organizations have a log analysis capability. It takes expertise to:

• Determine the organization's requirements
• Turn on proper logging accross the enterprise
• set up log collection

Generally, these things are not outsourced, but you can buy products to help with this. Loglogic is mentioned several times in this document, but we have a large number of choices including: ArcSight, NetForensics, HighTower, LogRythm and dozens more. But the real driver is when you analyze the logs. That requires a lot of expertise and the people that have that expertise are pretty expensive. That is the value proposition of log analysis outsourcing, consider this snippet from Informationweek:

"Kettering owns the network security equipment, but for the last two years it has had Symantec (NSDQ: SYMC) collect and analyze data from firewall logs. "We need to be concerned if someone is trying to do a port scan against our systems or if our network contains ad bots or spy bots trying to communicate out," says Bob Burritt, Kettering's IS network and technology manager. The ability to detect and avert downtime is crucial to any organization, but particularly a health care operation. Added incentive is the $1 million a day Kettering would lose if it couldn't bill or collect fees. Burritt declines to say how much Kettering is paying for Symantec's services, but he notes that outsourcing firewall log analysis saves as much as $150,000 annually, roughly the cost of hiring two full-time IT pros."[11]

The Computer Security Manager Bottom Line for Outsourcing Log Analysis and Storage

Many organizations avoid outsourcing security services because they are concerned that another organization might collect information about them, or they may want the sense of control. And, to be sure, there are potential pitfalls to outsourcing, but you will almost certainly save money. In addition, real experts on log analysis are hard to find, even harder to find than people who are truly expert in intrusion detection/intrusion protection and even if you do find one or two, what happens if they decide to leave for greener pastures.

===
1. http://www.reuters.com/article/pressRelease/idUS142783+09-Jan-2008+PRN20080109
2. http://www.counterpane.com/log-management.html
3. http://searchsecurity.techtarget.com/tip/1,289483,sid14_gci1158739,00.html
4. http://www.sans.org/training/description.php?mid=66
5. http://csrc.nist.gov/publications/nistpubs/800-92/SP800-92.pdf
6. http://www.linkedin.com/in/chuvakin
7. email interview, Dr. Anton Chuvakin
8. http://www.itl.nist.gov/lab/bulletns/bltnoct06.htm
9. http://www.sans.edu/resources/leadershiplab/352.php
10. http://www.softpanorama.org/Logs/Syslog/syslog_configuration_examples.shtml
11. http://www.informationweek.com/news/showArticle.jhtml?articleID=196604332