The 7 Deadly Sins of Performance Measurement and How to Avoid Them

Operational performance measurement remains an unsolved problem. Despite the relatively little attention it gets in the management literature, designing and using metrics to track and improve operating performance is one of the most persistent problems that organizations face. In my interactions with companies in virtually every industry, I scarcely ever encounter one that believes it has an effective set of metrics for their operations: manufacturing, customer service, marketing, procurement and the like. To be sure, companies do have measurements for these areas that they employ every day, but few managers or staff believe that these metrics are the right ones or that they help the company improve its performance and achieve its strategic goals.

This is remarkable for two reasons: First, operational performance measurement is so fundamental to basic operational management that it should presumably have been resolved a long time ago; second, in the last several years companies have developed much more sophisticated strategic measurement systems, based on such tools as the balanced scorecard, key performance indicators, computerized dashboards and the like. Nonetheless, among the hundreds of managers with whom I have discussed this matter, there is a widespread consensus that they measure too much or too little, or the wrong things, and that in any event they don’t use their metrics effectively.

The most striking manifestation of this problem is that many of the operational metrics that companies commonly use make little or no sense. I have found that organizations fall prey to a half dozen or so recurring mistakes in defining and using metrics, mistakes that seriously impede the relevance and usefulness of their operating measures and that help explain the widespread malaise about measurement that they feel. I call these the seven deadly sins of performance measurement, and, like the seven deadly sins of theology, they present grave dangers, if not to the prospects for the immortal soul, then to the prospects for superior business performance.

1. Vanity

One of the most widespread mistakes in performance measurement is to use measures that will inevitably make the organization, its people and especially its managers look good. As one executive said, “Nobody wants a metric that they don’t score 95 on.” This is particularly the case since bonuses and other rewards are usually tied to results measured in terms of performance measures. For instance, in the area of logistics and order fulfillment, it is common for companies to measure themselves against promise date — that is, whether they shipped on the date that they promised the customer. A moment’s impartial reflection shows that this sets the bar absurdly low — a company need only promise delivery dates that it can easily make in order to look good on this metric. Even worse, companies often measure against what is called last promise date — the final date promised the customer, after changes may have been made to the delivery schedule. It takes real effort not to hit the last promise date. Moreover, achieving good results on last promise date has no larger significance for company performance; it does not lead to customer satisfaction or any other desirable outcome.

A far better metric would be performance against customer request date — but achieving that goal would be more difficult and might lead to managers not getting their bonuses. When executives at a semiconductor manufacturer proposed shifting from last promise date to customer request date, they encountered widespread pushback and resistance.

A metals refiner had been using yield — the percentage of raw material that was turned into salable product — as a key performance metric, and everyone was very pleased that this figure was consistently over 95%. An executive new to the company made the observation that this figure glossed over the difference between high-grade and low-grade product; the refinery was supposed to produce only high-grade product, but poor processing sometimes led to low-grade product. The company then started to measure the yield of high-grade product and discovered that figure was closer to 70% — and a much more meaningful representation of the refinery’s real performance. Not surprisingly, this insight was not welcomed with great enthusiasm.

2. Provincialism

This is the sin of letting organizational boundaries and concerns dictate performance metrics. On the surface, it would seem natural and appropriate for a functional department to be measured on its own performance — after all, that is what its managers can control. In reality, however, measuring narrowly inevitably leads to suboptimization and conflict. For instance, one insurance company chief executive has complained that he spends half his time adjudicating disputes between sales and underwriting. Sales is usually measured in sales volume, which motivates the sales force to sell to any willing customer. Underwriting, however, is measured in quality of risk, which leads personnel to want to reject all but the best prospects. The transportation group in a retailer was measured in terms of freight costs. While ostensibly reasonable, this led the group to search out the best deals in shipping, even if this meant that deliveries to the distribution centers would sometimes be early and sometimes late — resulting either in out-of-stock situations or in chaos at the receiving dock.

3. Narcissism

This is the unpardonable offense of measuring from one’s own point of view rather than from the customer’s perspective. A retailer, for instance, measured its distribution organization based on whether the goods in the stores matched the stock-on-hand levels specified in the merchandising plan. It had a satisfying 98% availability when measured in this way. But when it thought to measure to what extent the goods in the stores matched what customers actually wanted to buy, rather than what the merchandising plan called for, the figure was only 86%. Another retailer measured goods-in-stock by whether the goods had arrived in the store; eventually, the company realized that simply being in the store did the customer no good if the product wasn’t on the shelf — and on-shelf availability was considerably lower than in-store availability. Many companies measure the performance of order fulfillment in terms of whether the shipment left the dock on the date scheduled. This is of interest only to the company itself — customers care about when they receive the shipment, not when it leaves the dock. A major computer systems manufacturer measured on-time shipping in terms of individual components; if it shipped, say, nine of 10 components of a system on time, it gave itself a 90% score. The customer, of course, would give the company a 0% rating since without all 10 components, the system cannot operate.

4. Laziness

This is a trap that even those who avoid narcissism often fall into: assuming one knows what is important to measure without giving it adequate thought or effort. A semiconductor maker measured many aspects of its order processing operation, but not the critical (to customers) issue of how long it took from the time the customer gave the order to the time the company confirmed the order and provided a delivery date — simply because it never thought to ask customers what was really important to them. An electric power utility assumed that customers cared about speed of installation and so measured and tried to improve that factor, only to discover later that customers cared more about the reliability of the installation date they were given than about its proximity. Companies often jump to conclusions, or measure what is easy to measure or measure what they have always measured, rather than go through the effort of ascertaining what is truly important to measure.

5. Pettiness

Too often, companies measure only a small component of what matters. A telecommunications systems vendor rejected a proposal to have customers perform their own repairs because that would require putting spare parts at customer premises, which would drive up spare parts inventory levels — a key metric for the company. It lost sight of the fact that the broader and more meaningful metric was total cost of maintenance — the sum of labor costs and inventory costs — and that the increase in parts inventory would be more than offset by a reduction in labor costs the new approach would realize. It is a basic axiom in the apparel industry that manufacturing needs to be done in Asia in order to lower manufacturing costs. Zara International Inc., the phenomenally successful Spanish apparel company, has recognized that the larger metric is product profitability, which does reflect manufacturing costs but is also affected by the timeliness of the product line and the volume of goods that need to be sold off at the end of the season. By doing production in Europe after the season has started and after new products have been tested in the field, Zara ensures that its fashions sell well and that it has little left at the end of the season, payoffs that more than offset the higher production costs.

6. Inanity

Many companies seem to implement metrics without giving any thought to the consequences of these metrics on human behavior and ultimately on enterprise performance. People in an organization will seek to improve a metric they are told is important, especially if they are compensated for it — even if doing so has counter-productive consequences. For instance, a regional fast-food chain specializing in chicken decided to improve financial performance by reducing waste — waste being defined as chicken that had been cooked but unsold at the end of the day and thus discarded. Restaurant managers throughout the chain obediently responded by driving out waste — in many cases, by telling their staff not to cook any chicken until it had been ordered. This had the unfortunate and perverse consequence of turning a fast-food chain into a slow-food chain; waste declined but sales declined even more. Similarly, the common practice of measuring warehouse inventory at the end of the month encourages the warehouse manager to clear goods out just before the end of the month and then rush to replace them at the beginning of the next month, thereby creating chaos and higher costs.

7. Frivolity

This may be the most serious sin of all; it is the sin of not being serious about measurement in the first place. It is manifested by arguing about metrics instead of taking them to heart, by finding excuses for poor performance instead of tracking root causes, by looking for ways to pass the blame to others rather than shouldering the responsibility for improving performance. If the other errors are sins of the intellect, this is a sin of character and corporate culture. An oft-heard phrase at one financial services company is, “The decision has been made, let the debates begin.” When self-interest, hierarchical position and loudness of voice carry more weight than objective data, then even the most carefully designed and implemented metrics are of little value.

These categories overlap and are related; a single metric may be evidence of several sins. A company that commits these sins will find itself unable to use its metrics to drive improvements in operating performance, which is the key to improved enterprise performance. It is hardly surprising that an enterprise that does not have the right measures of operating performance will be hard-pressed to improve that performance; bad measurement systems are at best useless and at worst positively harmful. As the old saying goes, “That which is measured improves”; but if you are measuring the wrong thing, making it better will do little or no good. Remarkably, these sins are not committed only by poorly managed or unsuccessful organizations; they are rampant in even well-managed companies that are leaders in their industries.

Such companies manage to succeed despite their measurement systems rather than with them. Why has such a vital area been ignored and allowed to drift into such comical errors? Most often, it is because senior executives have not demonstrated a serious commitment to operational performance improvement. Either because they are far removed from the operational fray or because they don’t recognize the opportunities for performance improvement and the central role that metrics play in it, too many leaders pay scant attention to metrics. As one manager said: “Our executives don’t take measurement seriously because they were turned off by accounting in business school.” As a result, too many companies simply do not give operational measurement the attention it needs. They follow the path of least resistance, using measures they have inherited from the past or the first metrics that pop into their heads. A serious commitment to performance improvement demands an equally serious commitment to designing and using effective operational metrics.

Redemption

There are four steps to redeeming an organization from measurement hell, purging it of the seven deadly sins and setting it on the path to sustained performance improvement. The first is to select the right things to measure, those aspects of organizational performance that are both controllable and important to achieving enterprise success; the second is to measure these things in the right ways, through metrics that capture their essence in usable forms; the third is to embed these metrics in a disciplined process for performance improvement, to use them for treatment rather than autopsy; and the last is to create an organizational culture and value system that encourages the disciplined use of metrics for ongoing performance improvement rather than regard them as threats to be feared or opponents to be vanquished.

Deciding what to measure. There are two related keys to ensuring that a performance measurement system is focused on the right things. The first is to emphasize end-to-end business processes, the cross-organizational sequences of activities that create all customer value. Processes transcend functions and other organizational units and are the mechanisms by which the myriad activities performed in an enterprise are integrated to realize results. Typically, an enterprise of any size has five to 10 primary business processes, each of which may be decomposed into a similar number of subprocesses. By focusing its measurement system on processes rather than functions, an enterprise helps create alignment and a common focus across disparate units; instead of each seeking to optimize its own unique metric, departments are encouraged to work together to improve the performance of the process(es) of which they are part. Thus, metrics for order fulfillment should dominate metrics for logistics or production or order entry; metrics for product development are more important than metrics for market research or engineering; and so on.

The second key to ensuring that the right metrics are selected is to determine the drivers of enterprise results in terms of these processes. An example will help clarify this concept.

A fashion retailer sought to increase revenues; since most of the management team had backgrounds in merchandising, executives immediately assumed that the key would be improving the company’s advertising program in order to attract more shoppers into the stores. However, the chief operating officer had recently joined the company from a different industry and was unwilling to jump to this conclusion. Instead, he led an exercise to determine what factors were most critical to the company’s success and to identify metrics that captured them.

A simplified version of this analysis goes as follows: Increasing sales requires attracting more shoppers into the stores and selling more to those shoppers; thus measures of traffic and of what is called conversion ratio (the percentage of customers who actually make a purchase) are important. But these are outcome metrics: desirable goals but not ones that can be achieved directly. The next step was to determine the drivers of these outcomes, the factors needed to get more customers into the stores and to increase the conversion ratio.

Advertising effectiveness and product quality were identified as the key drivers of increasing traffic and therefore as important phenomena to measure as well. The factors needed to increase the conversion ratio were ensuring that products were on the shelf (since customers can’t buy what isn’t there) and having enough salespeople available to help customers decide what to buy; thus on-shelf availability and customer coverage (the ratio of salespeople to customers) were recognized as important metrics as well.

In this case, the assumption that the key to improving revenues lay through improved advertising turned out to be false. When measured, customer traffic, advertising effectiveness and product quality were at levels that ranged from acceptable to high. The problem lay in the conversion ratio — not enough shoppers were becoming buyers. The root of this problem was twofold: Neither on-shelf availability nor customer coverage were as high as they should have been. These were the areas, not advertising, which needed attention. But how can on-shelf availability and customer coverage be improved? This is where the connection to the processes is made. For each of the factors that is measured, the processes that affect that factor must be identified; the factor becomes a key metric for each of these processes, and the improvement of this metric is to be accomplished through process management: effective execution, ongoing improvement and holistic redesign when necessary.

In this case, on-shelf availability was recognized as being shaped by the supply chain process and so became a key, overarching metric for that process; customer coverage was seen as determined by the employee scheduling process. (This approach has similarities with Kaplan’s and Norton’s strategy maps, except with a more operational focus and an explicit linkage to end-to-end processes.) Changes were made to each of these processes in order to improve these measures, which in turn increased the conversion ratio, which in turn led to the desired improvement in revenues. In particular, this analysis led to the recognition of a very deep-seated problem with the employee scheduling process. In the past, it had been driven by when employees found it convenient to work rather than when customers were coming into the stores. Consequently, there were too many people working on weekday afternoons, and not enough on weekends. The new metrics and the process redesign effort they spawned soon changed that.

Measuring the right way. Knowing what needs to be measured is just the first step; finding the right way to do so is the next. As outlined above, deciding what needs to be measured is something of a science; deciding how to measure, however, remains an art, since, in general, there are many different ways of putting a number on a phenomenon that has been determined to be worthy of measurement. For instance, how should customer satisfaction be measured? One common approach is through customer surveys. However, this is costly and slow; it is also often uncertain how well customer responses on surveys correlate with desired behaviors. Measuring complaint volumes may not capture the full spectrum of customer attitudes and is subject to manipulation — not answering the complaint line guarantees a higher reading of customer satisfaction. Measuring attrition and repeat buying comes too late to do anything about it.

The point is not that these or any other specific measures of customer satisfaction are good or bad, but that virtually every metric has some advantages and drawbacks, and that in
designing metrics, one must balance the following considerations.

Precision. A metric must be carefully and exactly well defined, so that there can be no doubt or dispute about it. Thus, “on-time delivery” can be interpreted in numerous ways, depending on what the target is (first promise date, last promise date, request date and so on) and what it means to be “on time” (on the date, within 24 hours, within 48 hours and so forth). It should come as no surprise that when a metric is not unambiguously defined, people will interpret it in ways that work well for them. For instance, the manufacturing organization at a consumer goods company used an imprecise definition of productivity as an opportunity to take downtime and turnover time out of the equation. The definition of a metric should also include the units being employed and the range and scale of the measurement.

Accuracy. In many situations, a company needs to measure what amounts to a Platonic ideal (customer satisfaction, advertising effectiveness, product quality and so on). Any actual metric will inevitably represent only an approximation of this ideal. It is necessary to keep in mind the distinction between reality and what is being measured and to close the gap between the two, subject to the limitations imposed by these other considerations.

Overhead. Organizations often fall prey to the temptation to construct a complex mechanism for calculating a metric, when a far simpler one would suffice. For instance, the fashion retailer discussed above needed a way of determining the conversion ratio at its stores (the percentage of shoppers who bought something). Various complex schemes were proposed, involving the use of radio frequency identification tags and various types of sensors. In the end, the company decided on the low-tech approach of hiring high school students to sit outside stores and count the number of people who went into the store and the number coming out carrying shopping bags. The more inexpensive and convenient it is to calculate a metric, the better. The periodicity of the metric — how often it needs to be calculated — must also be taken into account.

Robustness. The designer of a metric must be conscious of the extent to which the metric can be manipulated or gamed by people with something at stake, or the extent to which the metric can encourage undesired behaviors. At a telecommunications company, for instance, using call duration to measure the performance of customer service representatives led CSRs to rush through calls.

A particularly valuable tactic to avoid suboptimization, undesired behaviors and the manipulation of metrics is to insist on the use of multiple rather than single metrics. For instance, measuring just the speed of product development can lead to cost overruns; measuring just transportation costs can lead to missing promised delivery dates. Individual metrics must be used as part of a system of related measurements, rather than in isolation. Competing process metrics (such as speed, quality and cost) should be balanced against one another, and process metrics should be used in conjunction with narrower-gauge, functional metrics. Thus, a shipping department should be held accountable for shipping costs (over which it has full control) as well as for on-time delivery (which it does not solely control), even though — or perhaps especially because — these two metrics can conflict with each other. Should people object, they need to be reminded that the world is not unidimensional, that the enterprise must serve constituencies with conflicting objectives and that balancing multiple metrics rather than overdosing on one is what success demands.

Using metrics systematically. Even the best-designed metrics are of little value unless they are embedded in a disciplined process that uses them. Otherwise, they become afterthoughts, employed to assess blame or reward, but not really utilized to drive improved performance.

A prerequisite for such a process is someone to perform it. In other words, every metric must have one or more individuals who are personally responsible and accountable for it; these individuals must realize that it is their job to ensure that their metrics achieve the target levels that have been set for them. Since many of the metrics that really matter in an enterprise do not line up neatly with the existing organizational structure, responsibility for end-to-end process metrics needs to rest both with the process owner (a senior manager with overall authority for the process) as well as with managers of the various functions involved in the process. The old saw that one cannot be held responsible for what one does not fully control has outlived its usefulness; it is both reasonable and necessary to hold everyone with some influence over a metric jointly accountable for it.

The starting point for using metrics to drive performance improvements is to have a target performance level for each metric. These targets can be derived in a variety of ways. In some cases, customers will be explicit about the performance level they require of a process — how long they will find it acceptable to wait for confirmation of an order, for instance. In other situations, it is the company’s own financial requirements or competitor performance that dictate required performance levels. Whatever the origin of the target, the person(s) responsible for the metric must regularly compare the actual value of the metric against the target level. If there is a meaningful gap between the two, particularly on a sustained basis, then the source of the gap must be determined.

Broadly speaking, there are two kinds of reasons why some aspect of operations is not meeting the required performance level: a design flaw or an execution problem. It may be the case that the design of the operating process or system simply does not allow it to operate at the target level. For instance, an electric power company found that no matter how hard people worked, no matter what tools or training it provided them, it could not connect electric power for new customers in less than 180 days; the process for establishing such new connections was so fragmented that it inevitably required multiple iterations to converge on an acceptable solution.

The design of a process establishes an upper limit on its performance; no process can perform better on a sustained basis than its design allows. Managers may discover that a process was designed to meet lower performance targets, or was designed with a now-obsolete set of assumptions or has become so burdened with special cases and accreted complexity that its performance has degraded. In such situations, a holistic redesign of the process is called for. Conversely, even a well-designed process is not guaranteed to deliver the performance of which it is theoretically capable. The electric power company just cited redesigned its process so that new connections could be made in 20 days. However, poorly trained or unmotivated workers, unreliable equipment or other exogenous factors could create execution problems that would lower performance beneath target levels.

In such situations, redesign will not help. Rather, a disciplined analysis of the underperformance of the process must be used to highlight its root cause, so that appropriate remediation — training, personnel changes, equipment repair or any of a host of other activities — can be taken in order to solve the problem and restore performance to needed levels.

The fundamental point is that measurement is not a thing unto itself. The point of measurement is not to measure but to enable improvement and, as such, measurement must be an integral part of an ongoing program of performance analysis and improvement.

Creating a measurement-friendly culture. Even the best metrics and the best process for using them will not flourish in an inhospitable environment. In too many organizations, the disciplined use of measurement is not part of the basic value system. Metrics are regarded as matters of opinion or as tools for political infighting; or metrics and their use are perceived as frightening, because they are used for the assignment of blame and subsequent punishment. Often, metrics are seen as pettifogging details only of interest to “bean counters” and similarly small-minded folks. The folk hero in too many organizations is the executive who flies by the seat of the pants and uses intuition and guts to make decisions, rather than the measurement-oriented analytic manager who is scrupulous about establishing the facts. Even worse, the prevailing culture in many organizations is to pass the buck when a problem is identified, lest one be caught without a seat when the music stops. In such contexts, the deadly sins of measurement are an inevitability that no technical solution can prevent.

Creating a measurement-friendly culture is not merely a matter of producing some inspiring slogans and printing them on laminated wallet cards. Changing the basic value system of an organization is much more complex than that; however, despite popular wisdom to the contrary, it can be done, and relatively quickly at that. The challenge is that to do so requires the personal time and engagement of the most senior leaders of the organization; they are the only ones with the stature and the authority to undertake such a deep shift. They have at their disposal a range of tools to help them accomplish such culture change, and they must make use of all of them:

Personal role modeling. As Albert Einstein said, “Setting an example is not the main means of influencing others; it is the only means.” When senior managers themselves personally insist on getting and using carefully designed metrics, and when that fact is publicized throughout the organization, attitudes change very quickly.

Reward. Making the use of the right metrics a part of how managers are evaluated and rewarded gets their attention. Rewarding managers who make use of the right metrics, even if the subsequent outcomes do not meet expectations, and concomitantly not rewarding managers who achieve desirable results without the disciplined use of metrics, makes the point that metrics are important.

Implementation. Rather than just preach about the use of metrics, senior managers can ensure their use by deploying metrics-based improvement and decision-making processes, making them part of basic management development training and conducting postaudits of key decisions to ensure these processes were used.

Commitment. Public demonstration of executive commitment to meaningful metrics is a powerful tonic. For instance, affirming the importance of using metrics even when it might be expedient not to do so drives home the message that measurement cannot be an occasional thing.

Articulation. It can do no harm for executives to talk about performance improvement, fact-based decision-making and carefully designed and meaningful metrics, and indeed doing so can help integrate these other techniques. However, mere words are not a substitute for more difficult and demanding steps.

When all these techniques are used together, the results can be truly impressive. At a privately held logistics company, the senior executives were dissatisfied with the measurement system, despite the company’s apparent success in the marketplace. They determined that a key driver of business performance was filling orders accurately, on time and with all needed supporting documentation, and so established the percentage of such “perfect orders” as a key operating metric. They then proceeded to identify their five key business processes and to find ways to redesign them so as to increase this figure. They also engaged people across the organization in the effort to focus on the metric and improve it. Over a period of several months, they managed to increase perfect orders from its initial value of only 6% to nearly 80%. As a result, operations costs have been dramatically reduced, customer satisfaction increased significantly and margins enhanced.

The deadly sins of operational measurement are not just measurement problems; they are symptoms of deeper cultural shortcomings, of a lack of comprehension of what is important to enterprise success, and of a fundamentally unstructured approach to performance management and improvement.

Poorly designed metrics cannot be repaired on their own, but only as part of a systematic effort to link operations to business objectives and to implement a formal process for operational performance improvement. The result of such an effort is much more than better metrics; it is a better company.