Leadership Laboratory

Leadership Lab: Audit and Governance

This series includes essays on security audit and governance. Tone at the top is a crucial aspect of leadership. However, our primary repository for audit information is the SANS audit blog: http://blogs.sans.org/it-audit/

Case Study: The Role of IT in Operational Risk - Updated October 6th, 2009
Applied Intelligence Analysis of Networks - June 16th, 2008
The case for outsourcing Log Analysis - January 11th, 2008
Qualitative vs. Quantitative Risk Assessment - September 15th, 2007
The Auditor and the PMBOK: Re-examining the Audit Process - February 28th, 2007

Case Study: The Role of IT in Operational Risk

October 6th, 2009
By Stephen Northcutt
Version 1.1


CEO Bill Jones was tired; two days and long nights of damage control were taking their toll. Expenses had been rising for the last year and a half, but then the acquisition talks began and GIAC Enterprises, the acquiring company, didn't seem too concerned. Now, on top of everything else, a rival in the business has announced they are now shipping medical gowns equal to those Bill's company produces in price and performance.

GIAC Enterprises sent in an auditing team to do due diligence and investigate whether or not the company was a good one to acquire. Today was the day the groups were meeting to present their reports. The conference room could barely hold the management and due diligence teams. Chairs were literally touching each other. The cramped conditions didn't help an already tense situation.

The Manufacturing Report

Though the manufacturing folks are still producing the best operating room clothing in the hospital supply business, the due diligence audit team made a disturbing discovery today. They found a dumpster full of rejects, literally 1 in 15 non-permeable, non-sparking Operating Room (OR) gowns were getting rejected by QA. How could we not have known that? Until today, those gowns had been an exclusive. No one else had the know how to produce these.

"We may have uncovered how your competition caught up with you," said the lead auditor. "We had a meeting with a number of your floor managers to discuss how the competition could have mastered the production processes so completely." One of them asked, "Could it have been that interview guy?"

The auditor asked, "What interview guy?" The manager responded that a person was running around with a clipboard interviewing people about their security processes, whether they changed their passwords or wrote them down, and whether they felt they knew the security policy. "He said his assignment was to interview 200 people if I recall. Did anyone else talk with him?" Slowly all the managers raised their hands. Every one, except the CEO and the facility manager, had talked to the interviewer.

The auditor continued, "Did he have a badge? Who authorized him to come in?" All the floor managers could do was shrug.

The news from the auditors' reports on IT was just as bad. They were concerned that the desktop systems had been purchased one at a time, there was no standard configuration. Many users had local security administrator. The auditors chose 10 systems at random and connected to the web to run Microsoft Security Essentials and found Malware on 7 out of 10. The report on the servers was not much better.

The IT Audit Report

After the auditors gave their report, Bill replied, "We have never really focused on IT. We are a manufacturing company."

The lead auditor responded, "They are related. The IT operations staff seems to have the mindset that they are just there to 'do their job,' but if something is broken, 9 times out of 10 it isn't anyone's job."

The technical analyst from the due diligence team followed up with, "The script that reported rejects to the management dashboard apparently broke 2 years ago." The CEO listened in disbelief. The production line manager had been looking at the same value on his dashboard, a 1.8 reject rate for 2 years and neither he, or anyone else had questioned it?

The analyst continued, "The QA department had reported the worsening QA problem, but, apparently no one read the email. Part of the problem was SPAM. The SPAM email server was overloaded and failing open, when it was unable to keep up, it simply passed along all the email. Some employees had SPAM that accounted for over 50% of their email. There was a lot of non-business email as well: joke lists, recipe of the day, that sort of thing. Email just isn't working well as a business tool for your company."

The Business Impact

GIAC Enterprises CEO, Bob Gantry, reviewed the due diligence team's reports and instructed them to table the acquisition effort before the final meeting.

The lead auditor delivered the news in the conference room. "We aren't saying never, but we don't want to acquire your business in its current state. It looks like you have a double whammy," he explained. "You had an security failure through which your intellectual property was exposed, and your operations department needs to be completely retooled. We have no way of knowing whether you can get better as an organization or not. Most importantly, you have lost your leading position in the industry. Long term, it makes business sense for us to have an operation like yours in house. If you can turn things around, please give us a call."

So, the acquisition that looked so certain only a week ago was in shambles.

Fixing the Problems

Bill knew they needed to address the issues the auditors uncovered:

Defect rate

The company was in the business of producing product and if there is a high defect rate cost containment is impossible. If there is a defect rate of 1 in 15, there has to be a process problem. While it is not normally appropriate to terminate an individual for a process failure, Bill and the management team are strongly considering replacing the foreman for the non-permeable, non-sparking Operating Room (OR) gown production. That individual probably had become demotivated by problems over time to reach a point where they just did not care anymore. Clearly, process changes and shortcuts have entered into the system. Someone familiar with implementing a Shewhart cycle (also known as PDCA (Plan-Do-Check-Act) ) in a fabrics production environment is needed to get control of the defect rate. Lack of Security Policies and Training

The competitor took advantage of the lack of security policies and training when they sent a bogus interviewer to get key information from floor managers. By using some simple social engineering tactics, they were able to learn a great deal about the security of electronic files. As a result, hacking into the overloaded systems was easy. Having a policy concerning identification of visitors and what type of information employees can share with those who are not employees would have helped avoid this important information being compromised.

Moreover, the lack of policies concerning disposal of rejected product led to substandard pieces being exposed to corporate spies. When combined with information about industry secrets gained through interviews, having access to samples made it much easier for the competitor to capitalize on what Bill's company had learned through a great deal of trial and error. They were able to create their own product. Having a security policy regarding disposal of products would have safeguarded the hard work company employees had done.

Equipment Failures

A survey of the company's computer systems revealed that the servers were old, some of them still running P3s, and the cooling system was totally maxed out. The server room was OK, except on the hottest days. Although the IT department was aware of this problem, a solution was delayed while waiting on an electrician estimate for NOC cooling. Bill knew this situation needed to be addressed in order to protect the equipment and keep the company running even if the current system failed.

Currently the company has a 3 ton unit in the server room that also provides dehumidifying and heating, if needed. The cooling unit is currently right at capacity, and the server room sits at a delicate balance. Human bodies in the room dramatically tip the scale.

The IT group had considered four options:

  1. a larger system to replace the current one
  2. server cage mounted units
  3. supplemental ductless units
  4. a variable capacity additional unit

Options 1 and 2 were prohibitively expensive. Option 3 was determined to be too unreliable and would not have been controllable. As a result, the best solution would be to add an additional cooling unit to the room.

The unit selected was a variable speed unit providing both a 1.5 ton and 4 ton capacity. The lower operational capacity would provide enough cooling to handle the expansion the company still had room to accommodate. The unit was also sized to prevent the short cycling of both units if two large units were competing. In addition, the 4 ton capacity would provide a backup in case the main unit failed. The company could operate for a short period with the second unit while the main unit was repaired. Currently there is no backup for the cooling system in the server room. This solution would cost around $8,000, plus electrical--a small investment for both additional cooling and a backup system.

Ineffective Email Service

The IT staff had already started on a project to rebuild the email system. The current relay server was incapable of keeping up with the times when the SPAM load was very high. During these periods, SPAM was passed through without processing to reduce load on the server. The IT manager reported that basic filtering worked pretty well, until we ran out of CPU cycles. He also noted that the company was running into disk space issues, which would require more frequent archiving to combat.

He continued, "The current email system uses qmail, and is very convoluted after years of being patched and added upon. We've run into issues with adding the CipherMail box to the system."

Bill worked with the IT staff to develop a plan to resolve these issues. In addition, he realized that the company might need to institute a policy regarding personal email on company computers in order to protect the companies assets. It was pretty clear to Bill that even though money was very tight, be needs to allocate some money to add some IT talent at the appropriate compensation.

What Should Bill Do Now?

There are several ways that Bill could get his company back on its feet. We present a notional solution below that is just meant to get you started. We encourage you to create your own approach to how Bill should proceed. Submit your advice to Bill (please limit to 750 words) to Stephen Northcutt, Stephen@sans.edu, we will post the better ones. Please make sure to include a short one or two sentence bio and if you wish, send us a headshot picture.

The Next Steps, One Approach to Recovery

Jeff Lake - Approach to Recovery

Bill gathered his management team for a meeting he hoped would set his company on the right path. He asked what the team thought were the key issues that, if addressed, could turn the company around. They listed a number of items that needed to be addressed:

Essential Business Planning

Bill's team needs to go back to the basics to consider what the company does, how it does it, and its strengths and weaknesses. Once those are identified, they need to think through the work flow processes. Here are some important questions for them to consider:

The answers to these questions will allow the group to better understand why the company is in business and how each person integrates with others to make the company work. Moreover, with answers in hand, the management team will be more open to accepting suggestions for improved security because they more fully appreciate the key processes.

Regulatory Compliance

Because there is more and more legislation to protect companies as society adapts to the digital world, it is incumbent upon companies to ensure they are familiar with Federal, State, and Industry requirements. Aside from the basic security ramifications, knowing how to evaluate or audit compliance with basic regulations is vital. The management team should be able to answer the following questions:

Workplace Ethics

With a highly-skilled workforce, many things are possible. Managers can coax employees to increase their productivity levels through various techniques. They can also improve the appearance of their operations and yet still have employee burnout and poor morale. You can get a lot done if you just selectively apply regulations and procedures, but is it right? How do you combat selective justification for bending rules? How do you foster a culture of ethical and moral behavior when society provides a broad spectrum of situational ethics?

Just because you can do something does not mean you should do it. The management team needs to consider:

How the management team sets the boundaries for the acceptable standard of behavior and methods of operation establish a sense of corporate identity. Setting the right standards can lead to an impeccable reputation and that can be sustained through ethical business practices and well-trained personnel. It is important that the management team involves the employees in the business. If employees know what to do, how to do it, what effect they have on processes, and if they have a sense of ownership in the success of the business, the stage is set for success.