Does What You’re Measuring Matter?

When a measure becomes a target, it ceases to be a good measure.Marilyn Strathern, “Improving Ratings” in European Review

In the face of the current complex business environment, most businesses strive to seek the simplified so that they can gain clarity and make effective decisions quickly.

It’s this desire that has produced flawed metrics like the Net Promoter Score (NPS)—a single measure that supposedly predicts business growth.¹ Despite the desire, there is no holy grail metric: The original NPS study only verified past behavior, despite claims of being able to predict future behavior; and, there is enough evidence that should make any company hesitant to use it as an indicator.²³⁴

This desire for simple metric points to a deeper issue that often goes unrecognized: metrics are only stand-ins for the thing they measure; they aren’t actually the thing being measured.⁵ It’s like the speedometer on your car: the speedometer measures electric pulses created by magnets attached to the driveshaft and the speed of the pulses indicates the speed. The speedometer isn’t actually measuring the car’s movement. 

When a business relies too heavily on a metric as an indicator it can become a problem because people will naturally alter their behaviors to improve a metric, but they may not actually be improving what they’re trying to measure.

This bias has been called Goodhart’s Law (“any observed statistical regularity will collapse once pressure is placed upon it for control purposes) or Campbell’s Law (“[t]he more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor”).⁶⁷ Or, as one anonymous tram driver in the 1950s put it: “Wherever there’s a system, there’s a racket to beat it.”⁸

But, the problem often isn’t employees trying to game the system to make themselves look good, it’s more often the result of putting too much emphasis on a metric instead of understanding the system as a whole. When too much emphasis is placed on the metrics, as often happens in business, people will naturally focus on the metric instead of the thing that it was designed to attempt to measure—the thing that actually has the impact on the business.

A metric is only as useful as its ability to provide insight into the thing being measured. Placing too much stock in just the metric and not in the underlying changes can lead you down a dangerous garden path.

Are you making your metrics matter?

_______________________