Embrace Your Ignorance

Microsoft’s Ronny Kohavi, a pioneer in online experimentation since his days as director of data mining and personalization at Amazon.com, likes to challenge his Internet-savvy audiences with a little experiment. Kohavi, a Microsoft distinguished engineer and general manager for analysis and experimentation, shows screenshots of simple but real A/B experiments that Microsoft ran to test different Web design options. Some visitors saw the A version, some the B version — and Microsoft tracked which version yielded better results. Kohavi then challenges his audience to predict the real-world outcomes of the test: Did Option A win? Did Option B win? Or were the results a statistical dead heat?

Almost without exception, his audiences fragment; not even a rough consensus emerges. “Stop debating,” Kohavi advises. “It’s easier to get the data … Data trumps intuition.”

Data from experiments such as Kohavi’s should ideally encourage executive curiosity and humility. Unfortunately, that rarely happens. Too many executives worldwide would rather use their expertise to “think” or “intuit” what’s best for their customers than run even the simplest experiment. They take pride in their “gut.” That’s a huge managerial mistake.

Organizations may be confident they know their customers, but they’re very likely to be overconfident. Most executives aren’t nearly as smart, perceptive or customer-centric as they believe. Leading researchers, such as Daniel Kahneman, who received the 2002 Nobel Prize in Economic Sciences, and psychologist Philip E. Tetlock, have persuasively demonstrated that “experts” are particularly prone to predictive overconfidence.

For example, in a study where experts were asked to make predictions related to world events, Tetlock reported in a 1999 article that “across all seven scenarios, experts were only slightly more accurate than one would expect from chance.” What’s more, Tetlock found that the “experts who assigned confidence estimates of 80 percent or higher were correct only 45 percent of the time.” The overconfidence — arrogance? — of presumed expertise is counterproductive. Pride goeth before a fall. Data trumps intuition. Serious innovators take data seriously.

The better business experimenters have cultivated an unusual way of adapting to their demonstrably limited powers of perspective: They’re humbled.

Digital marketing and analytics maven Avinash Kaushik once wrote in his blog Occam’s Razor that “80% of the time you/we are wrong about what a customer wants.” Regis Hadiaris from online mortgage lender Quicken Loans observed in a 2008 interview that “in the five years I’ve been running tests, I’m only about as correct in guessing the results as a major league baseball player is in hitting the ball. That’s right — I’ve been doing this for 5 years, and I can only ‘guess’ the outcome of a test about 33% of the time!” Microsoft’s Kohavi reports that only a third of the ideas his company experimentally tested measurably improved the metrics they were supposed to improve.

Note the selection bias behind those numbers. Those aren’t “let’s throw something against the wall to see what sticks” results; smart people ran those experiments because they thought they’d be winners. That experimental experience and intuition don’t even match the predictive validity of a coin toss should be sobering. As Kohavi and his colleagues have observed, “We are poor at assessing the value of ideas.”

Consequently, the better business experimenters have cultivated an unusual way of adapting to their demonstrably limited powers of prediction: They’re humbled. But they make their humility empowering. They make a virtue of their fallibility. Humility invites data-driven exploration and innovation.

How so? Successful innovators have the courage of their curiosity. They run experiments not because they can confidently predict what the outcomes will be — but precisely because they can’t. They’re less interested in validating their hypotheses than in challenging their assumptions. They want to learn from — and with — their customers. So they literally and figuratively experiment with customer-centricity.

In today’s digital economy, presumptuousness is for losers. Winning the hearts, minds and wallets of customers requires more than empathy and good ideas. The simple truth is that, on average, testable hypotheses and simple experiments that reveal customer preferences are smarter investments in customer insight than intuitions and ideas.

One of the most intriguing evolutions in innovation that I see in business today is a commingling of “big data” and experimentation to achieve greater customer insight. Data volumes are enormous, and opportunities for cost-effective experimentation are vast. Serious innovators increasingly recognize that customer-centric success with big data requires lots of little experiments — many of which will disprove the innovators’ initial hypotheses.

Where big data and associated analytics suggest patterns and correlations that merit further exploration, smart executives exercise a kind of intuition different from expert overconfidence: They choose to act humbly. Instead of leaping forward with grand plans, programs and initiatives, these executives see an appealing invitation to perform simple experiments. They seek to create virtuous cycles of profitable insight between big data and little experiments.

Will the combination of big data and little experiments usher in a new era of customer-centric innovation? That’s my intuition. But let’s see what the data reveal.