The Problem With Big Data

Because of big data — a term that has come to refer to the immense amount of digital material we generate, store, and manipulate with increasing ability — managers can measure more about their companies and then use that information to drive performance. Need to heighten the productivity of your workforce? Big data can help. Want to analyze customers’ preferences and purchase patterns? Big data can do that, too. Looking for ways to cut costs and increase profitability? Big data: At your service.

But not all companies are flourishing in this new era. Small companies are struggling. Over the last three decades, the annual rate of new startups has fallen from 13% to less than 8%. During that time, the percentage of employment at companies with fewer than 100 workers has decreased by 5%. Meanwhile, big companies are thriving. The share of revenue of the top 5% of businesses has increased by 10% since the 1980s. Large companies also employ a greater share of the U.S. labor force: from one-quarter in the 1980s to about one-third today. What accounts for this discrepancy?

My colleagues — Juliane Begenau of Stanford University’s Graduate School of Business and Laura Veldkamp of Columbia University Business School — and I had a hunch that one of the likely culprits was big data. Our reasoning: Investors rely on big data to help them make smarter investment decisions. Because big companies produce more data relative to smaller companies, investors have more information to go on. This wealth of data, in turn, accelerates advances in processing speeds and computing and helps investors view these large companies as a less risky bet. As a result, big companies get more than their fair share of financing at better terms and are therefore better able to prosper. Smaller, younger companies, by contrast, receive less financing, which hampers their ability to grow.

To test our hypothesis, we developed a noisy rational expectations model to demonstrate how big data influences the evolution of company distribution. Our model calculates the different availability of data in small and large companies and then looks at how that data interacts with the growth of processing power.

To understand just how rapidly it has changed, consider these striking facts: Data processing power has grown about 20% per year since 1980, where our research begins. In 1992, the average computer processor speed was 0.05 GHz. In 2005, that number was 3.6 GHz.

Our model also analyzes the evolution of the cost of capital over time and calculates the risk and return trade-off of financing a company about which little is known. We find that improvement in processing power has favored the risk-return trade-off for large companies with a wealth of data, and their cost of capital has fallen considerably. The findings are clear: Big data disproportionately advantages large companies over small ones.

Intuitively, this makes a lot of sense. Big companies have more economic activity and longer company histories, so they have more data to process. Meanwhile, all the computing power in the world cannot inform an investor about a small company that has a short history and fewer disclosures.

The problem, though, is that as big data technology improves, large companies will continue to draw a more than proportional share of data processing and financing support.

Advances in information technology have already exacerbated this trend. Faster processing speeds have enabled investors to crunch ever more data — macro announcements, earnings statements, competitors’ performance metrics, export market demand, anything and everything that might conceivably forecast future returns — in a matter of seconds. More data processing lowers investor uncertainty and thus lowers the cost of capital for big companies. Smaller companies can’t hope to compete.

Three Recommendations for Preventing Big Data Inequality

So, what can be done to fix this situation? For starters, managers of small companies need to be made aware of this inequality and its impact on their ability to access capital. A shift in mindset is required. Startup founders and small business owners must begin to think about their data as a new class of economic asset — like gold or oil. They need to understand their data helps investors learn about and assess them — which, in turn, helps them raise financing at lower costs. An abundance of data is a valuable asset; a dearth of data is increasingly seen as a damning liability.

Second, investors need to know that their bias toward data-rich companies means they are missing out on some potentially lucrative opportunities. After all, a vast body of research shows that startups and small businesses are often more productive, more innovative, and have better growth prospects than larger companies.

Finally, policy could be potentially helpful here, too. While our model does not analyze the impact of policy interventions, it’s important that lawmakers recognize the fact that the explosion of big data does not equally benefit all companies. One possible remedy would be to create incentives that help small companies generate more data so they can attract financing.

The promise of the big data revolution is evident. But we must better ensure that all companies can reap its benefits.