Security Laboratory

Sec Lab: Security Products

In 1995 if you wanted a security product, you downloaded the source and compiled it on your Sun 3, today we buy supported commercial products: this series on the security lab is to introduce you to some of the products out there and, when possible, the movers and shakers that are part of the team that creates these products.

Interview with Eric Hines, CEO of Applied Watch Technologies

Oct 12th, 2007
By Stephen Northcutt
Eric Hines is an IDS specialist and CEO of Applied Watch Technologies, we certainly appreciate his willingness to share his thoughts with the SecurityLab.


Eric, what is Applied Watch, what is the itch that you are the scratch for?

Hi Stephen, thank you for the opportunity to introduce Applied Watch to the SecurityLab reading community. Applied Watch is solving a real pain in the market for an enterprise-grade management solution for open source software. The itch we scratch is the need for a sexy, polished user interface to these command-line open source tools, which previously struggled in the the enterprise. It's my belief that a fundamental shift is occurring within the enterprise towards open source software. We're seeing organizations in the enterprise/carrier space forklifting their expensive commercial network management and security solutions for open source - from removing Checkpoint for IPTables at the perimeter to decommissioning HP Openview for an enterprise roll out of Nagios. I'm not talking about small companies here either, Applied Watch only sells into the enterprise to carrier space. So the companies I'm referring to are large telcos, wireless companies, banks, and federal/military. The fact of the matter is the cost savings of open source is no longer possible for CXOs to ignore. Open source is now becoming a topic in senior management meetings. Applied Watch makes open source in the enterprise possible, initially gaining market penetration in the open source Snort management space as the alternative to Sourcefire. We have over 600 installations and are continuing to grow. The secret sauce behind Applied Watch is our Java-powered, browserless Dashboard and unique Agent-Server architecture that allows customers to use the free, open source version of Snort downloaded from Snort.org (Applied Watch does not distribute the Snort binary.) Our mission is to be the enterprise console for every popular open source tool relied upon in the enterprise, supporting Snort, Snort-Inline, LaBrea Tarpit (Sticky Honeypot), Nessus, ClamAV, and soon Nagios and OSSEC HIDS.


Thank you Eric, you mention open source, do you feel that enterprise organizations can safely use open source? How do they know they are not getting bad software?

I believe, from what Applied Watch has seen, that the open source landscape is evolving to include the enterprise perimeter and core network. Open source is no longer the "red headed stepchild" being used in secrecy by sysadmins to do their job. I think the days of concern over the quality of open source software are gone as the cost of COTS (Commercial off-the-shelf) software continues to go up and the overall quality and security of those products continue to decline. Whereas open source was once believed to be "less secure" compared to commercial software, I think companies are starting to have a change of opinion as their budgets continue to decrease and productivity is expected to increase. Do I think open source can be safely used in the enterprise? Sure, open source is just as safe as, if not safer than, COTS solutions due to the huge user communities that support and contribute to these open source projects. Microsoft proved to the market that closed source software is just as vulnerable as open source software despite the source code being hidden from hackers/crackers. Just because a hacker can't see the source code, doesn't mean it's more secure and history has proven that. How do they know they aren't getting bad software? I suppose the same concern can be raised in closed source software that can't be reviewed, so depending on how you look at it, wouldn't you agree that reviewing the source code is far more advantageous than not being able to see it at all? The NSA reviewed the source code for Snort and distributes it internally to the US military as an "NSA Approved" version of Snort. The only way that was possible was because of the source code availability of Snort. Whereas they wouldn't be able to do that with any other IDS/IPS solution. So I could argue that open source is actually more secure and safer than COTS solutions. We have no way to interview the developers a security company hires, nor whether or not any backdoors or trojans exist in the code. Who's writing our code and what does that code look like?


When I think of open source the first two programs that come to mind are Nessus and Snort, do you support these? What other software do you support?

Great question Stephen. Actually, yes, we support both Nessus 2.x open source and 3.x binaries as well as all versions of open source Snort. The Applied Watch Command Center is modularized, allowing us to create a support module for virtually any open source tool. Users can schedule and execute Nessus vulnerability scans as well as compare Nessus reports to track which vulnerabilities have been fixed since the last scan. Our mission is to add support for popular open source tools one at a time and do it well. Our support for Snort allows users to monitor Snort in real-time in a browserless, Java-powered Dashboard, manage and create policies for each Snort sensor, push rules to each sensor, and download new rules using our Wizard from Bleeding Threats or snort.org VRT subscriptions. Our support of Snort is expansive, even allowing customers to convert old Snort policies to new versions of Snort as rules and syntax changes from version to version. The Dashboard gives users a real-time heartbeat monitor that shows the Agent and Snort process availability. One thing you've heard me talk about here is our "browserless" Dashboard. Very early on, in the concept-stage of Applied Watch, I made a design decision to release the first commercial console for Snort that wasn't web-based. I felt as a user of ACID and Demarc's free solution, that the web browser just wasn't an appropriate platform for monitoring an enterprise roll out of IDS sensors. It's simply too clunky -- clicking on anchor link upon anchor link to finally get to a packet, Internet Explorer crashing, or it looking different in Firefox. I wanted Applied Watch to offer the first ever commercial fat client for managing Snort. Another design decision that has allowed us to win a significant number of sales of competing Snort-based appliance companies is our software version of the product. Take one of our largest customers for example. They had over 1200 Snort sensors, probably one of the largest Snort deployments I've ever seen. Both Sourcefire and Demarc lost that sale to us because they were able to take the software version of our Agent, place it on their commodity hardware and immediately be up and running with full management capabilities over their open source Snort installation. No company is going to buy a bunch of $30,000 appliances to replace their already-existing hardware when they can buy a software package to achieve the same thing.


LaBrea Tarpit? That is that nifty piece of software written by Tom Liston right? Can you explain what it does and why people might want to run it?

Yes, LaBrea is a phenomenal tool. Unlike traditional "honeypots", LaBrea is what's considered to be a "Sticky Honeypot" using TCP window sizes to actually stick a scanner or worm from being able to move to other hosts. There are rumors of people having CodeRed stuck in a LaBrea Tarpit for over 6 months. The Command Center can actually correlate between attacks on the network and those targeted at Tarpitted IP space from LaBrea. Our Action Taken column tells the user whether the packets were tarpitted, dropped, or logged.


Nice explanation Eric, we appreciate that. One of the problems in information security is TMI, Too Much Information, and I see that you have taken that problem on. What are three major ways the Applied Watch Command Center can help with the TMI problem?

Well, as I'm sure you're aware, Applied Watch acquired Demarc Security back in Q1 of this year and we're in the process of implementing their patent-pending Threat Index Engine into our Command Center software that actually assigns threat index confidence levels to each Snort alert. This TIC score, if you will, is calculated based on OS and service information knowledge of the target IP of the packet. I don't believe there is a single silver bullet for fixing the problem of false positives in IDS and IPS solutions. I think it has to be a multi-faceted solution that is primarily based on as much knowledge about the endpoint system as you can get. Sourcefire has attacked this problem with RNA, we're doing it using a combination of support for different open source tools that achieve the same thing. Applied Watch is focused on adding support for open source tools to achieve the needs of our end users rather than trying to develop our own proprietary IP to do it. We want to stick to the grass roots of our company and technology of supporting open source projects.


A subset of the TMI problem with Snort is false positives, what do you think we can do as a community to reduce the number of false positives?

I think one of the best things we can do is continue to improve the accuracy of our IDS rules, tightening them down and focusing more on the vulnerability rather than the exploit itself. I also think we need to continue down this path of endpoint intelligence by the IDPs. One example: if an Apache Chunking Exploit is detected against an IP address that the IDP knows is a Windows 2000 machine running IIS, it knows this is a false positive and should be ignored.

You'd be surprised by how many security engineers I've met who still want an IDP that they can just drop into their network and forget it's there. One other thing I continue to promote is spending time creating individual rule policies for each IDP an organization has installed. Perhaps configuring an IDP to monitor only Windows machines, another monitoring only Unix machines, etc. This of course relies heavily on network design and architecture, although I'd never recommend that an enterprise IDP roll out without the involvement of the network engineering staff for intel on network architecture.


One of the interesting problems we have in security is software flaws. What safeguards do you build into your product to ensure Applied Watch itself cannot be attacked and subverted?

Again, great question. Applied Watch has gone to great lengths to try and make the software as secure as possible knowing that it's a wrapper around other security tools. We've designed our Server to connect to our PostgreSQL database using local file descriptors rather than having PostgreSQL listen on a port.

I've always found web-based IDP solutions to be adding unnecessary attack vectors to the system through the requirement for a web daemon running on the IDP Server. Because our system is not web-based, our Java-based Dashboard is also inherently insusceptible to buffer overflow attacks. Other safeguards in our product, including the use of AES 256-bit encryption, also add a great deal of confidence to the security of our product. Though, as I've said in the past, no company is immune to having their product show up as an advisory on Bugtraq, but you do the best you can and respond as quickly as you can to identified bugs -- as you know, no software is perfect and fully secure.


One of the things that I like to do in an interview is offer a bully pulpit, a chance for you to speak about what you think is most important in information security, what really has your attention these days?

I'd like to spend this time talking about the new OSSEC HIDS project Stephen. This is actually something that has excited me a great deal lately and has a lot of my attention. The project's user community is continuing to grow and, to me, it is the next Snort for the HIDS/HIPS space. Daniel Cid is the project's founder and developer. This is a very exciting project and I urge your readers to check it out. It has been ported to almost all the major OS platforms. From the www.ossec.net web site, OSSEC is an open source Host-based Intrusion Detection System (HIDS). It has a powerful correlation and analysis engine, integrating log analysis, file integrity checking, Windows registry monitoring, centralized policy enforcement, rootkit detection, real-time alerting and active response.

Applied Watch prides itself in supporting the open source community. We're a big supporter of the OSSEC project and have also designed the OSSEC logo.


I think it is a challenge to remain both technical and to be a business leader, I know it is for me. You clearly have the technical part dialed in, what can you share about what you have learned in your role as CEO? What advice can you pass on to someone that is technical and getting ready to take on significant management responsibilities?

Yes, I suffer from the same challenge. I've definitely had to trade in the bash shell for KPI reports and Profit and Loss reports, but honestly? It's been a welcomed change. I'm probably the most unique security analyst you've ever met in being a HEX monkey who likes looking at packets as much as reading through AP/AR Aging reports, and who also enjoys responding to the daily changing climate of running a company. I don't know if you remember, but I was fortunate enough to take the very last GCIA class you taught at SANS in San Diego. (Laughing) Though, despite all this, I definitely do find myself struggling to remember things I used to know when I did nothing but intrusion analysis (Smile) However, as I'm sure you've found, the company and its employees rely heavily on us to stay away from packets and focus more on running the business.


Last question Eric, tell us a bit about yourself, what do you do when you are not sitting behind a computer?

Hmm, who is Eric Hines? Well, I'm a single father of a beautiful 3 year old little boy. I love snow skiing and have a weakness for suit and tie shopping :). I've been playing the guitar for over 12 years now, and when I'm not behind the computer I'm probably taking photographs with my Cannon Rebel. I prefer black and white photos over color.

Thanks for your time, Stephen and for giving me the opportunity to tell everyone what we're doing here at Applied Watch with open source.