Minding the Analytics Gap

In an increasingly data-driven business environment, many executives must make critical decisions based on analyses that use data and statistical methods that they do not fully understand. How can executives with limited analytics expertise become adept consumers of analytics under such conditions? This question has become an important management issue as senior executives increasingly recognize the importance of analytics to creating business value.

XL Group plc, a global insurance and reinsurance company based in Dublin, Ireland, is a case in point. Like others in the insurance industry, XL has long relied heavily on data analysis to understand and price its products. Actuarial science itself is rooted in using historical data to understand future risk and uncertainty. Across the insurance industry, companies have access to better data and more sophisticated methods of analysis than they did in the past; analyses of only a few years ago are no longer adequate to keep modern insurers competitive. In response, XL produces increasingly complex analytics, and demand for analytical insights progressively permeates the organization. According to Kimberly Holmes, senior vice president of strategic analytics at the XL Group, “An increasing number of managers must take action based on analytical results. But unlike the earlier adopters who embraced analytical approaches, these more recent adopters are not as well versed in the concepts, tools, systems and techniques of contemporary analytics. They are not comfortable making decisions based on analytical approaches that they do not fully understand. Yet they must still make these decisions.”

XL is far from alone. Our research — based on a survey of 2,719 managers in organizations from around the world — finds that the foremost barriers to creating business value from analytics are not data management or complex modeling skills. (See “About the Research.”) Instead, the number one barrier by far in this year’s survey was translating analytics into business actions — in other words, making business decisions based on the results, not producing the results themselves. One survey respondent described his organization’s top analytical challenge as “developing middle management skills at interpretation.”

To date, considerable attention has been focused on the tools, systems, methods and skills necessary to produce analytical results, with recent emphasis on making those results more accessible to managers.1 But access does not translate to understanding. Certainly, making analytical insights more digestible to decision makers can help yield better business results. However, the challenge of translating analytics into actionable insights remains. Achieving competitive advantage further requires that managers better consume and apply the analytics their organizations produce.

Consuming, Not Just Producing, Analytics

Managers clearly have more access to useful data than ever before. During the past three years, the percentage of our survey respondents reporting increased access to useful data has steadily risen. However, the results are less clear when it comes to whether or not managers feel they have all the data they need to make key business decisions. (See “There’s More Useful Data — But Is There Enough?”) Increased amounts of data open new opportunities for organizations to understand their business, their customers and their environment, driving adoption of sophisticated methods of wringing insights from data. But potential value is lost when data analysts do not understand the business and managers do not understand the analytical results — when there is a gap between the organization’s ability to produce analytical results and its ability to apply those results to business issues.

What can organizations do to ensure that this increased access to data translates into better decisions? Companies can begin by addressing this problem from both the production and consumption sides of analytics.

From the production side, much can be done to make analytics more consumable by managers. At the individual level, data analysts can learn more about the business; in fact, about a third (34%) of our survey respondents reported that their organizations train analytics professionals to understand business issues. Organizations can also systemically improve infrastructure and processes; improved data quality, for example, can make it easier to turn data into competitive advantage.2

From the consumption side, managers can also take steps to become savvier at understanding analytical results. In fact, managers and executives are working to become more knowledgeable about data and analytics: Many of our survey respondents this year reported that their organizations develop analytical skills through on-the-job (58%) or formal (23%) training. Almost half the respondents (49%) reported that their organizations train managers to make better use of analytics. Beyond training, other known steps include identifying trustworthy analytics professionals within the organization, requiring straightforward explanations and asking detailed questions.3

However, our research indicates that, despite their efforts, managers continue to find it difficult to keep up with their analytics colleagues4 for two reasons: burgeoning analytics sophistication and competing demands for attention.

Burgeoning Analytics Sophistication

Efforts to create competitive advantage are leading to increased analytics sophistication; producers of analytics continue to use more data and more sophisticated techniques. Survey respondents who reported greater use of complex analytics by their organizations were also more likely to report that their companies are gaining a competitive advantage from analytics. (See “Complex Analytics and Competitive Advantage.”) While, overall, our survey results indicate that complex analytics tends to be associated with competitive advantage, the relationship is not perfect and suggests that producing complex analysis does not, by itself, convey advantage.

From our survey data, we identify three levels of analytical maturity in organizations. We call these groups Analytically Challenged, Analytical Practitioners and Analytical Innovators. These groupings are based on questions that assess how well an organization (1) derives competitive advantage from analytics and (2) uses analytics to innovate. Analytically Challenged organizations generally rely more on management experience than data analysis and tend to lack data management and analytical skills. Analytical Practitioners tend to use analytics for operational purposes, have “just good enough data” and are working to become more data driven. Analytical Innovators are more strategic in their application of analytics, place a high value on data, and have higher levels of data management and analytical skills.5

These three groups of companies show considerable differences in how they use three types of analytics — descriptive, predictive and prescriptive. Descriptive analytics focuses on what happened in the past and why. Predictive analytics uses models to forecast the future, and prescriptive analytics goes further to provide guidance by evaluating possible scenarios. Our survey results indicate that Analytical Innovators far exceed Analytically Challenged organizations in the extent to which they embrace predictive analytics and prescriptive analytics. (See “Analytical Innovators Embrace Sophisticated Approaches.”)

Competing Demands for Attention

Furthermore, as analytics techniques grow more sophisticated, managers are often not able to focus on catching up with these developments. Although data-driven approaches are important, managers and executives, by definition, also have other responsibilities within the organization. One survey respondent described priorities succinctly as “financial crisis cancels analytics,” while another noted difficulty finding “time to address the issues rather than put out fires.”

As a global marketing company that helps a wide range of companies connect with their customers, Epsilon Data Management, based in Dallas, Texas, is in a unique position to appreciate the pressures on managers. CEO Andy Frawley explained that both the complexity and pace of change in analytics tools and techniques make it hard for managers to keep up. Observed Frawley, “The other dynamic that plays into this is, a lot of these analytics need to be in real time. It’s no longer the case where somebody would say, ‘I want to build a model to predict who is going to purchase a product or service from our client,’ and we’d spend six weeks building a model and another six weeks deploying it. That concept is sort of obsolete now. This stuff has to be happening all the time in real time, because that’s the pace at which consumers are now acting. And that drives a very different cadence.”

Sophistication in Production and Consumption

Using our survey data, we created measures of the analytics production and consumption sophistication of the respondents’ organizations. We based the measure of sophistication in analytics production on the responses to three questions. First, respondents assessed their organizations’ use of analytics on a five-point scale from “simple models” to “complex models.” Second, respondents reported how effective their organizations are at “using predictive analytics.” Third, respondents rated their organizations’ effectiveness at “using prescriptive analytics.” We aggregated these responses to create a composite score of the sophistication of analytics produced. (See “Two Aspects of Analytics Sophistication.”) Analytical Innovators tend to be more sophisticated producers of analytics than other organizations.

Similarly, we based the measure of sophistication in analytics consumption on the responses to three questions. First, we asked each respondent to what degree his or her organization “has the appropriate analytical talent to make good use of analytics.” Second, respondents assessed how well their organizations as a whole are applying analytics to key business issues. Additionally, respondents selected the three biggest gaps or challenges in analytical skills in their organizations. Responses of “translating analytics into business strategic/specific actions” and “turning analytical insights into business actions” indicate difficulties in effectively consuming analytics. We aggregated these responses to create a composite score reflecting the sophistication of the organization’s ability to consume analytics. As with sophistication in producing analytics, Analytical Innovators are generally more sophisticated in their consumption of analytics than other organizations.

As organizations become more adept at producing analytics — and as the complexity of the problems and analytics increases — managers must work to keep up. For these more analytically sophisticated organizations, the existence of a gap between the organization’s abilities to produce and consume analytics may actually signal greater opportunity to elicit business value from analytics. These Analytical Innovators continue to find opportunity — even more than in Analytically Challenged organizations — to consume the analytics they produce. Analytical Innovators, however, must work hard to make the improvements required to build or maintain competitive advantage.

Gaining Comfort With the Analytics Gap

When an organization’s capacity to produce increasingly sophisticated analytics outpaces managers’ abilities to understand, discomfort is created — managers find they must make decisions based on complex analytical insights that they do not yet fully understand. But, despite this discomfort, these managerial decisions must be made.

Kimberly Holmes of XL observed that “your biggest risk is doing nothing because insurers are increasingly using analytics. Our competition is using analytics. Our brokers are using analytics. And if we are not on the forefront of this, we’re going to get left behind and won’t be able to catch up.” But she also likened some decisions to bungee jumping, where “the minute I’m geared up and I’m standing at the edge of the cliff, I’m thinking, ‘I don’t want to do this.’ It matters that you’re actually willing to make the changes in your business and lead your underwriters to make a change in the business — which are going to be sometimes painful — to get that business benefit.”

What should managers and executives do to prepare for such uncomfortable leaps? Our research identifies five ways that managers are increasing their comfort in consuming analytics.

1. Bolstering Your Knowledge Base

Even if they are not able to become analytics experts, managers are augmenting their personal knowledge as a foundation to understand analytical results. Understanding of analytics is far from binary. Instead, it builds up from incremental concepts drawn from multiple areas such as statistics, machine learning, data management and information systems. While many managers cannot expect to understand advanced analytics topics, becoming conversant with the basics provides a starting point from which to build. Managers can familiarize themselves with concepts such as descriptive, predictive and prescriptive analytics, and they can read about what other organizations are doing with analytics. While managers may not understand the specifics of every technique, they can build an analytics vocabulary and shared framework. Michelle McKenna-Doyle, chief information officer for the National Football League, noted how difficult her job would be without some knowledge of football by saying, “I would have been at a huge disadvantage if I didn’t speak their language.” Similarly, managers are finding themselves at a disadvantage if they do not speak the language of analytics, even if not fluently.

2. Building Off Prior Experience

As with other decisions, prior experiences with making analytics-based decisions reduces discomfort. Our interviews identify four ways that this prior experience benefits in the context of analytics. First, experience builds trust in the producers of analytical results. Managers build this trust through either their own direct experience or by trusting others who have experience using similar analytical results. Second, repeated communication with analytical producers improves managers’ abilities to frame the right questions. Third, experience builds familiarity with your organization’s data (for example, in terms of data types, sources, quality and completeness). Fourth, later assessment of the accuracy or inaccuracy of prior analytical results helps identify areas where discomfort is appropriate or inappropriate.

3. Creating Analytical Options

Making high-stakes decisions exacerbates managerial discomfort with analytics. We find that managers are instead building on a series of small steps in using analytical approaches. For example, at XL Group, Kimberly Holmes keeps “the focus on incremental improvement,” where each decision is “to keep the train rolling with steady deliverables,” not on big changes. In general, organizations are finding that small decisions, followed by quick assessments through experimentation, help build comfort with analytics-based decisions.

4. Capitalizing on Domain Knowledge

Managers are also assessing the source of their discomfort: To what degree does it come from limitations in the analysis itself rather than limitations in the manager’s understanding? While managers may lack detailed understanding of analytics techniques, they typically have the advantage of a deep understanding of the business context.

Good analysis often finds something retroactively obvious. Even if the details of the analytics technique are not completely understood, the resulting insight should resonate with managers because it successfully reveals an important underlying business mechanism. Unexpected results may be particularly insightful if they foster additional inquiry.

5. Recognizing the Limitations of Models

Managers should use their knowledge of the business context to reconcile discrepancies in analytical models or understand how sensitive the analytical results are to variations in inputs. For example, at CVS Health Corp., a pharmacy and pharmacy benefits company based in Woonsocket, Rhode Island, extensive models support decisions about new store locations. Decisions based on these models can be uncomfortable because it is difficult to lower commitments after a location decision is made: The store must be built in order to assess the model’s predictions. Hartwell Hooper, director of store location research at CVS, described the modeling difficulties resulting from the company’s 2014 decision to stop selling tobacco products. All of the company’s store location predictive models were built with historical sales data that included tobacco sales. Through their knowledge of the changing business context, managers were able to identify this limitation in the analytical models and commission additional analysis to understand the potential effects of the elimination of tobacco sales on store location decisions.

Looking Ahead

Overall, the trend toward analytics and data-based decision making continues. Our report last year found that analytics was becoming a common path to value.6 This year again finds a significant majority (61%) of respondents reporting that their organizations derive a competitive advantage from analytics. However, that percentage was higher among survey respondents in 2012 (67%) and 2013 (66%) — perhaps reflecting that it is harder to gain a competitive advantage from analytics now that analytics use is widespread.

As organizations increasingly embrace analytics, our survey and interviews reveal a sizable gap between the production and consumption of analytics. Furthermore, this gap persists and may even grow as organizations mature analytically. Producers of analytics will likely continue to improve their ability to make more sophisticated analytical results, so managers need to find ways to become comfortable making decisions based on analytical results that they do not fully understand. While the exact nature of the discomfort will evolve, the necessity of managing, despite it, will remain.