Big Idea: Competing With Data & Analytics

Big Data Fatigue?

It may be everywhere now, but big data matters more than ever.

Randy Bean Reading Time: 4 min 

Topics

Competing With Data & Analytics

How does data inform business processes, offerings, and engagement with customers? This research looks at trends in the use of analytics, the evolution of analytics strategy, optimal team composition, and new opportunities for data-driven innovation.
More in this series
Randy Bean

Randy Bean, CEO and managing partner of NewVantage Partners, a data and analytics executive management consulting and advisory firm.

I was having lunch recently with the editor of a leading business publication, when I was asked, “Don’t you see a backlash against ‘big data’? Aren’t people growing tired of hearing about it?”

This seemed a fair question, given the sheer volume of articles and media coverage on the topic of big data during the past three years. But my response was a resounding, “No!”

During the summer of 2013, NewVantage Partners conducted a survey of Fortune 1000 C-suite executives from companies including American Express, CVS Caremark, JP Morgan, Johnson & Johnson, Kaiser Permanente, MetLife, Travelers and Wells Fargo, among others.

Our survey found that 91% of executives indicated that they have a big data initiative planned or in progress, with 60% reporting having an initiative completed.

Let’s consider three reasons why big data is becoming part of the mainstream now.

1. Big data is about all data, not just social media, unstructured or massive data.

I’m sometimes told by senior corporate executives that they don’t have a big data need because they are not focusing on social media data, unstructured data or massive data sets. This is a common misconception about big data. While much of the talk about big data focuses on the benefits and opportunities that result from new sources of data — including social media, sensor and visual data — most of the action among mainstream corporations is focused on integrating information from traditional legacy environments, like COBOL and mainframe data sources.

Integration of legacy data remains one of the main challenges for most corporations born before the digital era. When asked what kinds of data corporations planned to integrate using big data capabilities, most respondents to our survey responded that their focus was on customer transaction and financial data. For corporations, the ultimate ability to link behavioral, transaction and customer interaction data provides insight into the relationships between what customers say and what they do.

Similarly, big data is not exclusively about massive data sets. It is often assumed that big data is exclusively about capturing very large volumes of data. But big data is also about integrating more sources (“variety”) of data, which needn’t be massive. As Babson College’s Tom Davenport states in a 2012 blog article, “Even small data can improve your organization’s judgment.”

Big data is about all data — big and small, structured and unstructured, new and legacy. And, for mainstream firms, executive respondents to our survey indicated that integrating and analyzing data from existing data sources was their greatest priority. In short, any organization that has “a lot of data” of any size, shape, form or variety has a potential big data opportunity.

2. Big data will change the time and cost equation for all data applications.

For many mainstream corporations, big data is no longer an experiment. Some context and history may help explain why.

Big data refers to a set of data management technologies, such as Hadoop, that were first employed by social media companies including Google, Facebook and Yahoo to enable the processing of massive volumes of information in a timely fashion.

Speed is realized through the ability to shorten the cycle from data access to analytical results. This is the result of the ability to load “all of the data” for quick and easy analysis, foregoing time-consuming processes or data engineering and up-front hypothesis formulation. Big data makes it possible to get more done for less by lowering the cost of managing data in two principle ways:

  • Reducing the need for costly upfront data preparation and data engineering, which typically constitute 80% of the time and cost of data management, and
  • Using lower-cost big data platforms like Hadoop, built with open source software, which are typically a fraction of the expense of traditional database platforms. (These numbers are based on our work with top financial firms and their own self-reports about cost advantages.)

Big data is becoming an integral part of the mainstream for large corporations as they come to understand that they can, for the first time, make data of any kind available to business analysts in a timely and cost-effective fashion. For these corporations, big data means faster time-to-market and quicker response to customer needs and interests, and the opportunity to accelerate time-to-value through greater speed and agility.

As big data approaches gain-growing acceptance, many mission-critical operational and regulatory reporting applications will be candidates for migration. Higher-cost production processes will be migrated to lower-cost platforms. This will be an evolutionary process, playing out over the course of the coming decade, as firms establish an evolving data-management ecosystem that combines traditional data-management approaches and new big data approaches.

For companies on the forefront of using big data, this migration of mission-critical processes is already well underway. Because big data represents an inexpensive way to get fast results, the need and the demand will only increase.

3. Big data is a term that captures the zeitgeist.

Love it or hate it, big data is a term that has caught on. But forget the semantics; focus on the benefits. In spite of lingering misconceptions, big data will compel interest and drive business value for many years to come.

Topics

Competing With Data & Analytics

How does data inform business processes, offerings, and engagement with customers? This research looks at trends in the use of analytics, the evolution of analytics strategy, optimal team composition, and new opportunities for data-driven innovation.
More in this series

About the Author

Randy Bean is CEO and managing partner of NewVantage Partners, a data and analytics executive management consulting and advisory firm.

Tags:

More Like This

Add a comment

You must to post a comment.

First time here? Sign up for a free account: Comment on articles and get access to many more articles.

Comments (3)
carnot antonio romero
Good piece. Interesting, though, that you suggest that data preparation is less important in the age of big data... if anything I would expect quite the opposite. 

Granted, you don't do the classic data warehouse ETL, but every time you try to do something on Big Data you have to figure out how to transform it into something that exposes the information you really want to analyze-- filter out irrelevant records or fields, flatten it or structure it, pre-aggregate it... It's just that now you may need to do it over and over again for every project, what with 'schema on read' and all. (Of course, you only do enough of it to answer the question at hand rather than building the perfect data warehouse... so that's potentially a savings, and an opportunity for a shorter time to first answer vs. waiting on a big bang data warehouse that never finishes. At a minimum, it means spreading out of the costs of prep work across each project that consumes the data.)

We need to:
- simplify the transformation design work to where most business analysts can manage without IT
- help users figure out whether what they have is of adequate quality, i.e. fit-for-purpose, and how to get it there if it isn't already
- democratize the cataloging, curating and searching of the great big bags of bits
- automate the discovery of relevant data and relationships for the specific problem at hand
- enable the identification and re-use of data prep work from other projects where possible, without committing to pre-computing, materializing and constantly refreshing the outputs in case someone wants them

New practices will be developed, new tooling will facilitate all this and really trigger the trip up the Slope of Enlightenment. But in the meantime, many will slog, shoulder-deep, across the trough of disillusionment, a lucky few encouraged by a backhand's brush against some crystallized chunk sunk in the sump, hinting that a toe might soon find purchase on the Slope of Enlightenment... 

But enough of metaphor! Clearly I need sleep. See you all on the Plateau of Productivity...
Steven Gold
Hello,
Thank you for your most insightful article! I would like to add some general observations of my own, gathered from work experience and academics in a variety of key functional areas. Please consider this comment as a type of “discussion draft” – here, the goal is to help facilitate discussions about what Big Data is and how we can best utilize it. Yes, “Big Data” is receiving a lot of attention from Senior Managers. Data Scientists, and IT Departments alike. In some ways, Big Data is the confluence of (1) the huge amounts of data being generated every day from new data sources which may not have existed 20 years ago, (2) the dramatic reduction in cost to store this “new” data provided by the Hadoop File System (HDFS), and (3) the development/availability of new technologies and Hadoop Ecosystem tools for storing, managing, and deriving actionable insights from the data. While these technological developments are both impressive and helpful – from my perspective, they may not always be self-sufficient in their own right. For example, from my perspective, some of the most important aspects of understanding and deriving actionable insights from Big Data are (partial list): (a) Metadata. To what degree do we know what the data means and what it does not mean? To what degree can we incorporate qualititative factors in our metadata? To what degree are we able to update and modify (sometimes selectively) metadata as the meaning and/or the structure of the data changes over time? (b) From my perspective, I view the domain knowledge of the Data Scientist as one of the keys to deriving more reliable data insights. For example, when analyzing Federal Reserve Bank economic data, a Data Scientist that understands the complex interrelationships between multi-variable data sets may be better able to provide value in this type of analysis. Here, from my perspective, it is not only the application of inferential statistics, machine learning, heuristics, etc. The Data Scientist, or Subject Matter Expert, may be able to provide perspective and insights, adding value which might otherwise possibly be overlooked. Hopefully, given sufficient interest, we can discuss this and more via future comments.

Very Respectfully,

Steven Gold

MS, Information and Telecommunications Systems, Johns Hopkins University, Baltimore MD

MBA - Marketing, University of Missouri, Kansas City MO

CSO
Strategic IT Security, LLC
srg@strategicitsecurity.com
Doug Laney
Couldn't agree more Randy. At Gartner we have compiled hundreds of real-world, high-value use cases of Big Data & advanced analytics, and we're learning of more everyday...particularly as you say Big Data projects are going "mainstream." People often misconstrue Gartner's Hype Cycle. That is, as a technology or technology concept becomes post-peak and starts sliding into the "Trough of Disillusionment" such as Big Data is now, this merely means that the fast-followers and laggards have more difficulty with it than the early adopters--mostly due to cultural, organizational and leadership barriers. Of course this is also when the usual boo-birds and naysayers come out of the woodwork.

Also, note that we have long contended that Big Data is about more than just "volume". My paper from 13 years ago first introduced the three-dimensional challenges and opportunities of increasing data volume, velocity and variety (the "3Vs").  Ref: http://goo.gl/wH3qG. And by 2:1 our clients tell us that data "variety" is the biggest challenge. 

Cheers,
Doug Laney, VP Research, Gartner, @doug_laney