Security Laboratory

Sec Lab: Security Products

In 1995 if you wanted a security product, you downloaded the source and compiled it on your Sun 3, today we buy supported commercial products: this series on the security lab is to introduce you to some of the products out there and, when possible, the movers and shakers that are part of the team that creates these products.

What to Look for in Log Management Solutions, an interview with Chris Petersen of LogRhythm

Apr 25th, 2007
By Stephen Northcutt
Chris Petersen from LogRhythm has agreed to be interviewed on the very important topic of log management. Thank you Chris, we really appreciate your time.

What is true log management? When a vendor says they have log management, what do they mean by that?
The two major elements are the logging and the management. First, logging is the collection of the information in its many formats, and through a variety of interfaces, to ensure it is reliably collected and not modified in transit. There are many means of capturing log data, some more reliable and secure than others. Since Syslog uses UDP, which by design is an unreliable network protocol, you will likely lose log data. Syslog does provides a great agent-less means of collecting logs, but is it reliable enough in all cases? If absolute reliability and perhaps encryption are important, an agent might make more sense. Of course, if it is a Cisco router or switch, an agent isn't an option so Syslog is the best you can do. However, I think it is important to understand the reliability options when collecting logs. If a more reliable mechanism can be used for collecting compliance relevant log data, you may want to consider the more reliable method.

The management piece is the management of information; that is, once I have collected it, what am I doing with it? Number one, it has to be prepared for analytics so it can be reported on in near real time and historically. However, you also have to consider how you will be able to find this data if it is required as part of reporting requirements in the future. There needs to be an effective archiving system for organizing the data so it can be automatically and easily brought back on-line. A good archiving system should employ compression and digital integrity verification mechanisms. It should also organize the information so it can be easily backed-up, taken offline, and destroyed when and if applicable.

For long term retention, I wonder if log management vendors will give customers the ability to store the data outside of the log management solution and recover it - that is the critical piece that is missing from some other solutions. Some SIM/SEM vendors feel that backing up their database is log management, but what is in the database? Raw log data, cooked normalized data. Even if you have raw log data, will you be able to bring it back years later? What if the schema changes during an update? What if the vendor goes away?

Chris, you seem to take this whole collection and recovery issues seriously. Why is it important?
How the information is collected will gain increasing importance in the years to come. In the not-so-distant future, auditors and regulators will put more emphasis on how you collected it; they will want you to use a reliable technique. When you think about log management, you need to think in terms of years and billions of records. When you get billions of records, managing this in a database gets tough, but if you store the data in an orderly directory, it is easier.

What about Syslog NG, would that be something the auditors would find sufficient?
That would be acceptable, a TCP based Syslog. I hope to see wider vendor adoption of this in the future. Unfortunately today, the vast majority of Syslog is still UDP, and I suspect this will be the case for years to come. Some log management vendors rely heavily on Syslog, but, while that lets them get product in the field, it is not reliable enough for the long term. Eventually, auditors will begin asking the questions: How is the data collected? Is it reliable? Do you have it all?

Splunk is getting a lot of press with the full text search. Can you comment on this? It seems that most vendors in the log management space have the same capability. And, how important is a full text search?
We and most other log management vendors have had text-based search all along, we just haven't marketed it as well as Splunk, I think we just took this capability for granted. However, unlike Splunk, we also have normalized data. When you are doing a full text search, you may want to contextualize what is being searched for against other values, you don't want to be looking for something brute force that is a million rows back. So, normalized data searches can be combined with full text searches to yield more powerful results.

I was sitting with a guy at breakfast and his solution could take as much as a full day for a search and report. What are some strategies that vendors can use to optimize searches and reports?
That is not good - an hour is not good. Part of the answer is to distribute the data; with multiple log managers you can parallelize the search using all of them, and there is a balance of how much data to keep in the database, but there is a cost for the storage. In many cases, people are searching for the meta data: when did they log in, to what server, is this an anomaly? You want raw logs, but you can pull information from the raw logs. Also, if you look for unique events (say a dropped packet, the src port, the dest port, if you get a repeat of the same IP, same src, and dest, you can start a count, doesn't take up that much storage, it is fast to search for, and lets you do trending. So, this way you can do this search very quickly against a data warehouse and when you know that the raw log you are looking for exists, it is worth the compute power to go after finding the raw logs.

Chris, what questions do you suggest buyers should ask vendors that we haven't already covered, to ensure the product they get is effective and useful for their company?
Here are a few to start off with, Stephen:

• What are the mechanisms they support to get data into the system, and are they reliable?
• Will they support what you want to collect today, as well as what you will want to collect tomorrow?
• What is your archiving solution? How is it stored? What is in the archive? If I lose my server, do I lose my archive too? Can I write the archive off to another system or another site?
• Reporting. Ask to see the reports - bring in some folks from business units, look at the packages the vendor has and try to make sure they are what you need. This is very important with SOX and other regulatory standards; is the report sufficient to meet the auditors' needs?
• Before talking to the vendor, work with the business units to figure out what information they need. Then, ask the log management vendor to show you that it is possible to get that data.
• Technical support. Especially in the first year, you are going to be asking your log management system questions every day. Can it give you the answers you need? And, when you can't figure out how to get it from the system, what facilities does the vendor have to help you get the answers you need.

If I were to buy a solution today, what could I expect for tech support?
You should be able to expect a basic solution to include software maintenance and 8 hour-a-day email tech support

Some vendors offer remote management services where the vendor literally manages the platform, keeps it patched and updated, and makes sure it is fully operational. This is important because Security purchases it while Operations manages it; and, if it is not a priority for Operations, does Security have assurance that the system is working correctly? Something like this is important in the mid-sized market.

You might also see if your vendor provides remote assistance services to help you with routine and/or sophisticated areas of managing and using their product, for example, creating custom reports and alerts.