How to Make Data Experiments Powerful

The power of experiments comes from the insight they provide into causal relationships. Simply put, by randomly manipulating only a single focal variable, we can assume that any observed changes are likely due to the manipulated variable, and not to something else.

Experiments based on data can be particularly powerful for organizations, especially if they easily augment managerial intuition and exploit unique data.

Data-based experiments are not new. Harrah’s Casinos provided early examples of how organizations could add confidence to organizational decisions, but at that time, many companies resisted aggressive testing. Change is difficult, particularly for experiments where some level of underperformance is inherent in the design. By definition, an experiment that shows differences between an indicator of performance will mean one group underperforms the another. For example, it can be difficult for managers to test a change that their intuition tells them will increase customer satisfaction or sales, because it means not following their intuition and reducing satisfaction or sales for a control group.

Now, there may be less resistance as organizations see the successes of others. Data-savvy organizations are setting up platforms to promote experimentation throughout their organizations. Experimentation can be “virtual research centers” that allow scale in R&D.

For example, Edmunds.com is a car shopping website with 20 million unique visitors every month. They built their own testing platform called the Website Testing Framework, on top of their extensive cloud-based data infrastructure. Phil Potloff, chief digital officer of Edmunds.com, says it allows all of their managers to ask “WTF (Website Testing Framework)? What’s going on with this test?” The framework supports dozens of simultaneous tests against three key performance indicators: user engagement, ad impression delivery rate, and lead volume. Questions can be big, such as evaluating an entirely new vehicle appraisal tool. Or tiny, such as the color of a single button on the website. Potloff reports that the framework has “really changed the way that we interface with analytics and how we make changes.”

WASH Laundry is another example of a company scaling experimentation. WASH Laundry manages laundry facilities embedded in 75,000 locations. More than 7 million people per week use laundry facilities under their management, generating a massive amount of data in addition to a lot of clean clothes. They’ve also just built an experimentation platform to support decision making.