The Downside of Real-Time Data

If information is power, you might assume that getting information more often should be powerful, too. Not so fast, say two business school professors.

“In many situations, real-time data comes in on a continuous basis, and then you, as a decision maker, have to decide which data is information and which is pure noise,” says Jayashankar M. Swaminathan, Kay and Van Weatherspoon Distinguished Professor of Operations, Technology and Innovation Management at the University of North Carolina’s Kenan-Flagler Business School. “That’s not an easy task, which is what this study shows.”

Swaminathan and Nicholas H. Lurie, assistant professor of marketing at Georgia Tech’s College of Management, conducted studies of undergraduate students to investigate how the frequency of information reports affects decision making. The researchers discovered that receiving information more frequently led to worse decisions, particularly when there was more “noise” — that is, random fluctuations — in the data.

“Is Timely Information Always Better? The Effect of Feedback Frequency on Decision Making,” an article forthcoming in Organizational Behavior and Human Decision Processes, describes the first study and establishes the fundamentals. Seventy-six students played a game in which they were retailers stocking the shelves with a perishable hypothetical product. Different sets of participants placed orders for periods analogous to either every day, twice a week or once a week. The participants were then shown the actual demand during that period for the product, and the demand fluctuated randomly within a predetermined range. The students were also shown their resulting profitability figures for the period. After that, they were asked to place orders for the next period, over the course of two 30-day games. No unsold inventory could be carried over from period to period.

Lurie and Swaminathan found that participants who received reports and placed orders daily had lower profits than their peers who got reports once or twice weekly. To put it another way: Even though all the participants received the same granularity of information — daily sales — those who got the information every day, as opposed to every three or six days, made worse decisions. This was particularly true when there was a high variance in actual demand.

What seemed to be happening is that students with daily information were more likely to give too much weight to the previous day’s data in making their decisions, rather than looking at a longer time period. The participants who saw data in groups of three or six days tended to factor more variation in daily demand into their ordering and thus were more inclined to smooth out demand in their estimations.

The authors confirmed the shortsighted-ness of daily report readers in a subsequent study. In this one, they made the record of each participant’s past sales reports available and tracked how participants accessed that data before placing their next orders. Although participants could access their history across previous rounds, regardless of feedback frequency, they tended to limit their information gathering to the most recent data. In other words, daily report readers generally did not take the time to reread reports for previous days before placing their orders.

Because the desire for real-time information flows is not limited to sales and production environments, the tendency to give too much weight to recent data points has implications in many business contexts, particularly those marked by volatility. Think of stock market traders, for example, who might trade on the basis of a stock’s momentum over the course of a day — never mind a week, a month or a year. Furthermore, the greater the variance in the data, the authors say, the more likely it is that more frequent reporting can skew decision making, because each individual data point becomes less representative.

The findings are not necessarily an indictment of capturing real-time information, however. Of more importance is how the data are presented to managers. “The way you present information drives behavior,” says Lurie. “If you take information and break it into pieces, you could end up with totally different behavior than if you present it as a group, even if the underlying data is identical.”

There are ways to mitigate the hazards of examining data too frequently, the authors suggest. For example, “If we had given people moving averages as well as the most recent information,” Swaminathan says, “I think they would start to think about deviation rather than just the most recent numbers.” And people’s decisions would be better for it. Moving averages are a simple way of implicitly incorporating a history of orders into a single figure that allows the latest data to be contextualized and the variance identified.

Techniques such as statistical process control charts serve a similar purpose by filtering data and identifying exceptions, so that managers aren’t focusing solely on the latest data. “It’s not like we’re saying you should scrap your real-time system,” Lurie says. “We’re saying use caution before rolling these things out and simply giving managers real-time data.” So the power of real-time information lies in its context, not its frequency.

For more information, download a version of the paper at http://mgt.gatech.edu/directory/faculty/lurie/pubs/lurie_and_swaminathan_feedback.pdf or contact Lurie at nicholas.lurie@mgt.gatech.edu or Swaminathan at msj@unc.edu.

— Larry Yu