Columbia’s Veldkamp: Big Data Has Hard Limits

The future belongs to Big Data.

From Wall Street trading floors to Amazon’s huge knowledge base on customer preferences to Tesla’s self-driving software to Major League Baseball’s analytics-driven front offices, Big Data—which, when connected to machine learning and/or artificial intelligence, aggregates massive datasets to analyze patterns of behavior and performance—appears to offer unlimited upside. As of 2015, information and communications technology were responsible for an estimated 6.5% of global GDP and some 100 million jobs.                   

As more and more data becomes accessible, there should be no limit to increases in the productivity users get from it.

But a recent paper by Poets&Quants’ Professor of the Week, Laura L. Veldkamp of the Columbia Business School, along with Maryam Farboodi of MIT Sloan School of Management, questions the “sky’s the limit” conventional wisdom about Big Data.  In fact, the research says there are real limits to its potential gains, suggesting that its usage will produce “diminishing returns” in the long run.

‘DATA-DRIVEN GROWTH WILL GRIND TO A HALT WITHOUT GAINS IN NON-DATA PRODUCTIVITY’

”Just like capital accumulation, data accumulation alone cannot sustain growth,” Veldkamp and Farboodi write.  “Without improvements in non-data productivity, data-driven growth will grind to a halt.”

The working paper, whose latest version was published last October, is called “A Growth Model of the Data Economy.”

Veldkamp and Farboodi begin by challenging the assumption that growth in the amount of data is equivalent to idea growth or technological change. The key features of data, they write, “are that it is user-generated and that it is used to predict uncertain outcomes.” But even a model that theoretically has perfect foresight doesn’t produce infinite profits.  

‘DATA IS A MEANS OF REDUCING UNCERTAINTY’

Why? “With perfect forecasting, zero operational mistakes, profits are large, but not infinite….Data cannot sustain long-run growth because data, like all information, is a means of reducing uncertainty….,” Veldkamp and her co-author write. “Information has diminishing returns because its ability to reduce variance gets smaller and smaller as beliefs become more precise.” In other words, the more data you accumulate, the less value future data will add.

“Unless a perfect forecast gives a firm access to a pure, real, limitless arbitrage, the perfect forecast generates finite payoff,” the authors continue. “Information has diminishing returns because its ability to reduce variance gets smaller and smaller as beliefs become more precise…Without any other source of growth in the model, data-driven growth, like capital-driven growth, eventually grinds to a halt.”

That “other” growth, of course, comes from economic activity, which generates more data points for researchers and practitioners to analyze and incorporate into their forecasts. But the data itself does not drive growth; that’s the result of economic activity and better tools to aggregate and analyze all the data it generates.

CAPITAL HAS LIMITS AND SO DOES BIG DATA

“The more productive capacity the data is matched with, the greater are the gains in output,” Veldkamp and Farboodi write. 

In fact, they think the use of data is more comparable to how firms employed capital during the Industrial Revolution than to the technological innovation of the 21^st Century post-industrial age. “When economies accumulate data alone, the aggregate growth economics are similar to an economy that accumulates capital alone,” the researchers note.

Like capital, data helps managers apply innovation to the real world and amplify its effect. And just as capital has limits, so does data, even Big Data, Veldkamp and Farboodi conclude.

Laura Veldkamp, 44, is the Cooperman Professor of Finance and Economics at Columbia Business School. Her research focuses on how individuals, investors, and firms get their information, how that information affects the decisions they make, and how those decisions affect the macroeconomy and asset prices. Lately she has examined the impact of the data economy. She teaches international finance to MBAs and finance theory and information frictions in finance to PhDs.

Veldkamp earned her bachelor’s degree in math and economics from Northwestern and got a PhD in economic analysis and policy from the Stanford Graduate School of Business. Having taught at NYU Stern School of Business for 15 years, she joined the Columbia faculty in 2018.

See all of our Professors of the Week