Why It Pays to Be Where the IT Talent Already Is

U.S. businesses are in the midst of a data-driven management revolution. Organizations capture enormous amounts of fine-grained data based on social media activity, radio-frequency identification (RFID) tags, web-browsing patterns, and mobile phone usage. Analysis of this data offers the promise of insights that will revolutionize managerial decision making.

In many cases, however, this type of data analysis has outpaced existing technological capabilities. Many organizations have been left wondering about the economic impact of their big-data technology investments. Are those investments translating into productivity returns?

One particular challenge enterprises face is difficulty in acquiring talent with the technical skills necessary to support big data tools. For instance, Sears Holdings’ early adoption of Hadoop, a framework for distributed processing of large data sets, was initially hampered by a shortage in people skilled in Hadoop as a platform. As demand for big data technologies grows, so do the problems of finding sufficient skills. The result: Talent shortages could limit the rate of productivity growth.

Labor Markets and Technology ROI

To understand the value that organizations are deriving from big data investments, I examined regional differences in the supply of workers and the skills required for new information technologies — especially during new technologies’ early periods, when there are few channels for acquiring these skills. Much of the value of technological investment, after all, is determined by the supply of professionals who can translate technologies into business outcomes. This variation may explain why some organizations unlock value from new IT innovations faster than others. I also examined how labor markets have specifically shaped early returns on investment (ROI) in one big data technology: Hadoop-based systems.

By studying returns on Hadoop investments that are concentrated in select labor markets, such as California’s Silicon Valley, we can observe that the more Hadoop expertise there is in the labor market, the easier it is to hire skilled Hadoop employees. At the time when I collected my data, more than 30% of workers with Hadoop skills were employed in Silicon Valley — a region that accounts for just 4% of total IT employment in the United States. Skills for more mature technologies were much less geographically concentrated.

It turns out that when know-how is scarce, organizations that invest in new IT or R&D derive significant benefits from the related investments of other organizations. As technologies mature and complementary skill channels emerge (through, for instance, university degree programs), the importance of labor market “spillovers” from the investments of nearby companies usually declines.

I used LinkedIn data to distinguish the human capital investments companies made in emerging data technologies versus investments in mature data technologies. The most robust productivity estimates from 2006 to 2011 show that Hadoop investments were associated with 3% faster productivity growth for organizations with significant data assets and in labor networks with significant aggregate Hadoop investment.

On the other hand, the estimates indicate that outside of these Hadoop-intensive labor markets, there were no measurable returns on Hadoop investments. In contrast, for mature data technologies, such as SQL-based databases, technical skills are widely available and the related investments of nearby companies are not significant. These findings formalize the notion that returns on companies’ Hadoop investments are increasing the investments of other organizations in their local labor market.

It’s also interesting to note that while companies with the largest Hadoop investments were mostly in IT industries, more than 30% of Hadoop investment was in non-IT industries, including finance, transportation, utilities, and retail. And that’s growing.

In sum, these findings suggest that geography, corporate investment, and channels for technical skill acquisition are all important factors in productivity growth rates during the spread of new IT innovations.

Trade-Offs for Managers to Consider

What does all this mean for managers? Although Hadoop investment appears to be associated with higher productivity levels in data-intensive industries, the analysis underscores the trade-offs managers of data-intensive organizations face: They must balance the benefits of extracting greater value from their data using emerging big data technologies against the higher costs of acquiring the required expertise in a tight labor market.

For managers who choose not to incur the expense, investments in traditional database systems — for which skills are widely available — may remain more effective. Alternatively, managers can wait. Big data technologies are maturing, and the channels for talent to acquire complementary skills, such as university programs, are expanding. Managers must also consider that acquiring complementary skills is not the only obstacle to successful big data use; organization-wide changes to existing data assets, management practices, and data governance may also be needed. As with many innovative practices, instituting these capabilities often requires organizational changes to complement data-driven technologies.

From a high-tech labor policy perspective, the findings suggest that access to complementary skills in some markets is associated with performance advantages for early adopters of big data technologies. At the same time, as this complementary know-how becomes more widely available, it should erode the productivity advantages experienced by companies in these markets.

Policies to accelerate big data know-how, such as those promoting business analytics courses, can narrow inequality in complementary skills across labor markets. But if there are significant lags in this process, organizations in data-intensive labor markets will continue to experience faster productivity growth than others.