[Return To Selected Articles]
May/June 2004 - By Joe Murphy
The Measurement Challenge (Part 1): Introducing the Deep Dive
Measurement is essential for an effective compliance program. More and more we are seeing that government agencies and company management want to know whether our compliance efforts are working and having an impact. Those
doing compliance work need to be able to demonstrate that the compliance program is more than a paper exercise.
How can we respond to this expectation? How do we measure what we are doing? There are growing efforts to measure compliance programs using things like surveys, focus groups, self-assessments and questionnaires. These all have their place, but they also come with some serious limitations. Each depends on the good faith of the person providing the information. One need only
picture an Andrew Fastow or a Bernie Ebbers participating in one of these exercises to see their limitations.
The existing devices also tend to focus on what is readily quantifiable. There is, in fact, a major risk that any push toward measurement will, in turn, push people to measure what is most measurable, and use the tools that lead to the most quantifiable results. It is certainly easier to measure how many employees say they feel good, bad or indifferent than is it to measure
whether anyone is actually engaged in accounting fraud or price-fixing.
On the other side of the spectrum, the compliance audit is a tool that involves rigorous reviews, including testing the veracity of what employees say. Audits, however, come with their own baggage. Although I have personally championed audits in the compliance field for two decades, and government officials and others have sung their praises, seldom have I seen their use outside of environmental compliance and
other traditional audit areas. Companies will certainly audit financial records, I-9 forms, and environmental conditions. But there appears to be an enduring resistance to a measurement device that is perceived as negative, and, by some, as a green-eyeshade checklist exercise. Opponents usually agree that audits dig deeply, but they view auditing as too narrow and limited. What seems to be missing is a measurement and evaluation device that fits the special needs of the compliance field. That
gap leads to the idea of the "deep dive."
What is a ‘deep dive’?
The term ‘deep dive’ is not alien to business, but it has not found its way into common parlance in the compliance field. It occurred to me that the concept belonged in the compliance lexicon when I heard it used by former General Electric CEO Jack Welch. As I understand it, a deep dive would be an intensive, deep focus on, and on-site review of, a segment of the business. In Welch’s parlance it may
be strictly a management device, but the idea is also viable for compliance purposes.
My interest in this subject began when I first did compliance field audits. Based on my 20 years as an in-house lawyer, I am a believer in the value of going out to the field. In my experience, it is often difficult for people in headquarters to know what is happening in the marketplace, in production, or in remote offices. My own experience is that such audits invariably turn up surprising
information. People in the field can be very clever in adjusting to their environments and pursuing ‘opportunities’ that are not always in the company’s best interests.
In the past, these field exercises may have been described as "compliance audits." When conducted the right way, they give the compliance staff the ability to find out what people in the company are actually doing. As a compliance person there is great value in immersing yourself in part of the business for a
period of time. Although the compliance audits I performed were often targeted at specific compliance risks, such as antitrust, they were also open-ended enough to allow me to respond to whatever might come up.
Intensive business reviews are not the only way to measure what is happening; an audit or other review is just one tool under item 5 of the federal Sentencing Guidelines’ seven-step framework for an "effective" compliance program. However, these types of reviews are
an important ingredient for any program in order to have the right mix of controls and measurements.
Why a deep dive?
If we have surveys and audits, why is there a need for a new concept? Surveys are certainly a valuable and necessary tool. As Win Swenson notes in Kaplan, Murphy & Swenson’s Compliance Programs and the Corporate Sentencing Guidelines (Clark Boardman Callaghan; 1995 & Annual Supp.), surveys "can provide statistically valid data to help management
determine if specific problems are widespread and systemic, or localized to a particular business unit." Surveys can give you trends and serve as red flags for potential trouble areas. Negative survey responses from the West Coast office may be a critical signal that further review is urgently needed. Discontent expressed in focus groups may signal that a particular office has deeper ethical issues.
However, what surveys measure is mostly perceptions, not what is actually
happening. They provide averages and trends, not specifics. Moreover, it can be difficult to test the legitimacy of what people are telling you in the survey responses. In the worst case, survey respondents can even game the system—if they want to rattle management’s chain, or they do not want the compliance people bothering them, for example. Surveys also do not burrow into an operation to look at key risk areas.
An example of this limitation can be seen in a recent report on the
Boeing compliance program. It was noted that a startling 98 percent of employees remembered the company’s ethics training and were aware of the company’s helpline. This would certainly seem to be reason for management to have confidence in the program. Yet this same company had its CFO ousted for breaking an important company rule, and had a business unit violate rules on using competitors’ trade secrets, resulting in a debarment order for that unit’s government work. Boeing had also, years
earlier, been subject to a record-setting fine for export control violations. Criminal conduct is not conducted by a majority vote of employees. It can occur when there are other program weaknesses, such as missing checks and balances in the system. A survey may completely miss those structural defects.
When surveys do reveal weaknesses in a business segment—morale problems, lack of commitment to compliance, management problems generally—a deep dive can be exactly the right tool to
use. It zeros in on what is actually happening in that part of the business and helps determine what needs to be done. Unlike an investigation, it does not search for individuals involved in specific wrongdoing; it constitutes a broader review of a troubled business unit.
Why not an audit?
If surveys are too broad to complete the picture, then why not conduct an audit? An audit, too, has certain inherent limitations. Understanding these weaknesses begins with the accounting
concept of "audit," and the perceptions attached to the term. Audits tend to be associated with a ‘check the box’ mentality. The auditor looks for the 10 things on a list, and success or failure is based on completeness. If something is not on the list, it is not checked. In the compliance field, this is analogous to someone checking the deck chairs on the Titanic, but not seeking out deeper problems such as bad rivets that could affect seaworthiness. The traditional audit may count
accurately, but still miss important matters because they are not on the list.
The audit concept also suffers from another flaw. While surveys are viewed as information gathering, audits have a policing connotation. The auditors come to find trouble. Even people who believe in auditing (myself included) hope never to be visited by an auditor. Auditing has an apparently indelible association with policing and distrust. This makes it a hard sell, especially in areas like antitrust that
do not seem to be "check-the-box" exercises.
When we consider the nature of the compliance function, it can be difficult to picture an all-purpose measurement device. As is evident from the Enron-era cases, compliance must deal with corporate crime—willful, deliberate violations of law and ethical standards. It also deals with things that may be hidden. And it deals with the unexpected. Companies face huge risks from things that it would never occur to them to test for.
Conventional measurement tools tend to focus on what is already known as a risk area. The deep dive, however, goes outside of this box. It looks for trouble, in effect.
The deep dive approach also has another advantage—the enormous value of getting headquarters people out to the field. A case could be made for having every headquarters manager—no matter the position—spend some time in the field. In the compliance area, field work helps ensure that compliance people are in touch with
reality. It can improve their effectiveness. There can be real value in requiring all compliance staff, including the subject-risk area compliance people, to participate in a certain number of these visits.
(This is the end of the first part of "The Measurement Challenge." The second part, which will run in the July/August issue of ethikos, will discuss how to conduct a ‘deep dive.’) o
© 2004 Ethikos, Inc. All rights reserved.
[Return To Selected Articles]