Why Your Company Is Probably Measuring Social Media Wrong

In our 2014 report on the state of social business, the single biggest factor associated with a company’s social business maturity was whether and how they collected and used data from and about their social business efforts. Yet I suspect that many companies’ approaches to measuring their social media efforts are wrong.

Standard data analysis techniques are often based in linear methods. These methods essentially try to determine whether a change in one variable is associated with a change in another — whether more of X leads to more of Y. For example, a company might seek to identify whether increased mentions of its brand in social media leads to higher sales or greater brand awareness. Managers are comfortable and familiar with this simple, direct line of thinking.

Social media environments, however, can be far more complex than these simple, linear relationships. Interactions on social media perhaps best understood as a complex adaptive system. These types of systems are often difficult to measure using traditional data analysis techniques, and exhibit four distinctive characteristics.

Nonlinear effects

Social media may exhibit a “Goldilocks effect,” where more of X leads to more of Y up to a point — but when X is “just right”, adding more of it stops having an effect or begins to have a detrimental one.

For example, we studied retention rates in an online community and found that the community operated best when there were moderate levels of people joining and leaving. Too much turnover meant that the community never developed an identity or direction, but too little turnover meant that the community became too rigid and insulated from new ways of thinking. A moderate amount of turnover was “just right,” leading to the best outcomes.

The takeaway: It’s never as simple as you think it is. Managers must be wary of oversimplifying the effects in social media environments. More complex relationships may exist, which require more sophisticated thinking and measurement.

Incremental effects

The impact of small changes on larger systems is often illustrated by the so-called Butterfly Effect, in which a butterfly flapping its wings in one part of the world sets a sequence of events in motion that leads to a hurricane in another via incremental effects. In the same way, some dynamics in one part of the social media environment — even seemingly minor ones — might have unexpected, potentially significant effects in another.

This mechanism might be sociological. We found that as participants left one community and joined another one, the knowledge that they had gained in their earlier efforts was transferred to the new community. Just as bees aid pollination by moving from flower to flower, so participants spread knowledge from one online community by moving to another.

This mechanism might also be technological. Small technical changes to a social media platform can have significant implications on user behavior. For instance, one colleague found that activity on a social media platform completely dried up when the platform began displaying the profile pictures viewed by its users. People had been using the platform to view pictures of other users surreptitiously, but when the new feature made that behavior public, it was entirely eliminated — and many users no longer had a reason to use the platform.

The takeaway: Small factors can have big impacts. Managers should be careful of the influence of unexpected or unknown effects in social media environments. More of X may lead to more of Y, until the seemingly unrelated Z changes both variables — or the relationship between them — in unexpected ways.

Emergence

Complex adaptive systems can exhibit patterns of order from relatively simple organizational rules. The classic examples here are flocks of birds or schools of fish, both of which appear to move as a synchronized whole while actually obeying relatively simple rules, such as staying a certain distance from others or moving toward visible light.

For example, we found that some online communities exhibit elements of Tuckman’s (1965) theory of group development — where groups develop through distinct stages of forming, storming, norming, and performing — even though the group did not have any formal leadership or stable membership that were previously thought indispensible. Familiar patterns simply emerged as the group’s interactions matured over time, even though there was no intentional effort to do so.

The takeaway: Social environments evolve over time — sometimes rapidly. Managers should recognize that the effects in social media communities could change over time. More of X may lead to more of Y at one point in time, but the story may be different in the future.

Dynamics

Complex adaptive systems can exhibit feedback loops where outcomes can spiral in ever-intensifying dynamics. For example, we found that in user-generated content communities, more content attracted more users, who then created more content, which in turn attracted still more users.

These types of feedback loops can lead to rapid growth cycles that are difficult to manage. Such feedback loops are thought to be a major cause of the demise of Friendster, a predecessor of Facebook, when they could not solve the technical requirements of a rapidly growing user base. They can also lead to accelerating death spirals that are difficult to recover from, as a departing user base leads to less content and fewer users.

The takeaway: Watch for sudden expansions or contractions in activity, and adjust to compensate. Managers should recognize that effects in social media communities can increase or decrease rapidly over time and must constantly recalibrate their measurement tools accordingly. X may lead to Y, which may also lead to X again.

Social media platforms provide unique opportunities to measure the behavior of customers and employees that can yield powerful insights for managers and marketers. Yet if appropriate measurement techniques do not account for the features of complex adaptive systems that characterize many social media environments, these analyses can lead to erroneous insights and bad decision making. The data and analyses enabled by social media are only valuable if they provide an accurate picture upon which managers can act and make decisions.