Online Reputation Systems: How to Design One That Does What You Need

Image courtesy of Flickr user rcade.

Some of the most exciting new ideas in organizations are based on harnessing the collective intelligence of crowds and communities. Think of user-generated content platforms, open source software, crowdsourcing, and knowledge markets — just a few examples of ways that large numbers of loosely networked users are invited to create or evaluate products, content or ideas. This is the “social web,” the interlinked virtual universe that to so many executives seems to offer the irresistible promise of providing something — ideas, work, decisions — for (almost) nothing, if only they could manage it right.

But the first step toward smart management of the social web is to understand something paradoxical about it: The new platforms may be all about harnessing crowds and communities, but in the end, those crowds and communities are nothing but a sum of individuals. And your company’s social web efforts will succeed only to the extent that you are able to attract good individuals, motivate them to perform good work, and empower them to get to know and trust one another enough to collaborate toward the end goals of the community.

The question is, How do you do that?

The answer : By capitalizing on the motivational power of reputation.

The best websites know that. (Even poor ones know it — they just don’t manage it as well.) Accordingly almost all social web platforms have some form of member profile feature, a set of pages where one can find out about individual community members. Member profile pages provide a space where community members can write about themselves, and their interests. In addition, profile pages often include statistics about a member’s activity, testimonials from other members and sometimes a score or rating that reflects a member’s accomplishments and status within that particular community. Other members can use this information to decide how much to trust a member, how much credibility to assign to his or her postings or whether to engage in a transaction or collaboration with him or her.

These features are parts of online reputation systems. Reputation systems are ubiquitous but often undernourished — and badly designed — components of social web platforms. Yet they play a crucial role in building trust, promoting quality, improving collaboration and instilling loyalty. More than almost any other aspect of a social web platform, reputation systems are the puzzle piece that can make the difference between success and failure. They’re the unsung heroes of the social web.

To name a few examples: eBay, a trading community of virtual strangers, would not have been possible without the reputation system that promotes honesty and builds trust by enabling buyers and sellers to rate each other. At the same time, eBay’s reputation system’s poorly designed has done an inadequate job of deterring fraud and, over time, has required eBay to put in place several additional monitoring mechanisms. Amazon’s product reviews feature is rendered both more useful and more credible by the presence of a scoring system that allows readers to vote on a review’s usefulness and uses such votes to point customers to the most useful reviews and thereby reward the most successful reviewers. Yelp, a youthful urban reviews community, similarly uses what is essentially a reputation system to help users better interpret opinions that are inherently subjective. On the other hand, Digg.com, a social news filtering site, instilled unnecessary competition among its members through a poorly designed reputation system. This ended up promoting collusive behavior to the detriment of the site’s quality and eventual impact.

Over the past 10 years researchers and practitioners in diverse fields have studied reputation systems, arriving at important results.

Key Decision #1: What are the key business objectives of your reputation system?

Before you even approach the website drawing board it is important to understand and prioritize the goals that a reputation system will serve in your social web platform, both from the perspective of your organization and from that of your users.

Reputation is a summary of one’s past actions within the context of a specific community, presented in a manner that can help other community members make decisions with respect to whether and how to relate to that individual. The concept of reputation is as old as society itself. Reputation exists and is propagated within human communities whether a central system mediates and facilitates the process or not. Small, tightly knit communities arguably do not need central reputation systems, since frequent interactions and gossip ensure that relevant information is known to all.

The need for a central system increases with the size of the community and the lack of frequent interaction among members. In web-based communities with hundreds or thousands of members, where most members typically know each other only virtually, some form of a reputation system is almost always essential.

Reputation is most often associated with the notion of trust. Although trust-building has historically been an important role of reputation, today’s web-based reputation systems serve a surprisingly broad set of objectives that includes the following:

Build Trust. Encouraging “good” behaviors and discouraging “bad” ones within the context of a site is perhaps the most obvious and widely discussed role of reputation systems, best exemplified by eBay.

Promote Quality. In the majority of social web platforms the quality of user contributions tends to be very uneven: a small number of users typically supplies the bulk of quality contributions. Reputation systems can enhance such systems by helping to recognize and feature high-quality contributors. This provides an incentive to contributors to try harder (so that they can be recognized) while helping users easily identify quality content.

Facilitate Member Matching. This objective is important in settings where members vary widely in interests and tastes and where the quality of contributions is characterized by a high degree of subjectivity. In such settings reputation systems can help users assess how similar or compatible they are with each other so that they can decide how much to trust someone’s posting or whether to initiate collaboration.

Sustain Loyalty. Since reputation is usually based on activity performed within a single community or web system, it constitutes a powerful form of lock-in and can be used strategically to increase user loyalty and decrease attrition. Once users have built a reputation on a site, they will be reluctant to defect to a competitor since they would then have to build their reputation from scratch. In an increasingly competitive environment this dimension of reputation mechanisms must not be overlooked.

All four objectives are relevant to the design of most reputation systems. Different social web platforms, however, assign different priorities to each of these objectives. For example, building sufficient trust so that buyers can feel comfortable enough to send their money to sellers they have never met and, very likely, will never buy from again, is arguably the primary objective of eBay’s reputation mechanism. On the other hand, the most important objective of Amazon’s reviewer reputation mechanism is to induce members to contribute well thought out, high-quality reviews and to identify (or, filter) the highest quality reviews. Inducing top-quality contributions is high on Yelp’s agenda as well. However, because the majority of Yelp reviews are subjective, it is usually difficult to reliably assess a review’s quality. What is more important is to provide tools that can help readers make their personal assessment of a review’s credibility and how well it fits their view of the world. Finally, Xbox Live is an example of a system where user loyalty and retention is the overarching (business) objective of its reputation feature.

Key decision #2: What information should be included in your user’s reputation profile?

A fair reputation system should be based on a user’s activity within a community. It should focus on reporting summaries of a user’s actions or ratings of a user’s community contributions. It is usually OK to allow users to post a few things about themselves. However, in almost all cases, reputation systems should not allow other users to post direct feedback on an individual, unless it is tied to a specific action or community contribution; allowing direct rating of individuals opens the door to gaming, blackmail and more.

Users typically engage in numerous and varied behaviors within the context of a site. Effective reputation systems must carefully choose which aspects of user behavior to track and report on. Keeping in mind the business objectives that the reputation system is intended to serve (Key Decision #1), there are three aspects to this decision:

Which actions are most relevant to the reputation system’s users? For example, if the key objective of the system is to help users decide whether a seller is honest, then keeping track of the seller’s percentage of completed transactions is a very relevant indicator, whereas keeping track of a seller’s own purchase history is less relevant. On the other hand, if a system’s key objective is to help users determine whether a reviewer has similar tastes to their own, keeping track of a reviewer’s purchase history might be a very relevant indicator.

Which user behaviors are desirable? The mere act of publicly keeping track of someone’s actions can encourage or discourage their incidence (depending on whether the action in question has a good or bad connotation). Therefore, if a site wants to encourage the volume of contributions, it might consider keeping track on reputation profiles of the number of reviews posted or the number of comments posted. On the other hand, if a site wants to encourage the quality of contributions it might want to hide information about contribution volume and keep track of how other people rated a particular contribution instead.

For which behaviors is it possible to obtain reliable information? The moment a reputation system decides to highlight some aspect of behavior, individuals are bound to try to game the system to their advantage. It is, therefore, important to choose metrics that are reliable and difficult to manipulate. The main choice here is between internally generated (first-hand) information and (second-hand) feedback provided by others. Yelp’s reporting of the volume of a user’s posted reviews or the size of one’s friends network are examples of first-hand information. In contrast, eBay’s reliance on asking the buyer and the seller what they thought about a transaction is an example of second-hand feedback. Relying on internally-generated information about user actions is generally preferable as it tends to be more reliable. Unfortunately, such information is not always available, or its collection might require the development of costly additional infrastructure. For this reason reputation systems must often rely on second-hand feedback provided by others. When doing so, one must put in place mechanisms to limit manipulation and gaming.

Key Decision #3: How should reputation information be aggregated and displayed?

Reputation mechanisms employ a variety of methods for displaying outputs. These fall into three large categories:

• Raw activity statistics. Examples: number of reviews posted, number of transactions completed.
• Scores and distinctions. Examples: star ratings (such as Amazon reviews), numerical scores (eBay’s feedback score, TopCoder’s user rating), numbered levels or named member tiers (World of Warcraft’s player levels, Slashdot’s “moderator” and “meta-moderator” tiers) or achievement badges (eBay power seller, Amazon Top Reviewer).
• Leaderboards and other methods of displaying relative user rankings. Examples: the list of top Amazon reviewers; Epinions’ author popularity ranking.

The choice of an aggregation and display method is very important because it can determine, a) the extent to which the reputation mechanism makes a judgment versus allowing users to make their own judgments, and b) the extent to which the presence of the reputation system can create competition among users.

Displaying raw statistics of a person’s activity within a community is perhaps the most neutral method of summarizing someone’s reputation. Social network statistics, in particular, have been rapidly rising in popularity in many contexts. The advantage of using raw statistics is that the reputation system thus makes minimal judgments and allows users to draw their own conclusions. The disadvantage is that the burden of interpreting these quantities falls on the shoulders of the user, who must be familiar enough with the environment to draw the proper conclusions.

The use of some form of explicit score, such as star ratings, solves this problem by immediately communicating to users whether somebody’s performance along some dimension is “good” or “bad.” This method helps users more easily digest information but also makes an explicit judgment about quality. It is, therefore, best in settings where there is a commonly agreed upon notion of what “quality” means.

Leaderboards and rankings go a step further as they not only imply judgment but also indicate a user’s standing relative to everyone else. They introduce a pecking order among users. This can have a very strong impact on a community’s culture and behavior. Supporting direct comparison of users against one another sometimes increases incentives to contribute. On the other hand, such direct comparison instills a culture of competition which might end up being disruptive in a number of ways. First, obsession with rankings might lead some users to manipulative and counter-productive behavior. Second, users who don’t make it to the top ranks might feel resentful and induced to exit the community. Experience has shown that in almost all cases, prominently displayed rankings or leaderboards lead to more trouble than benefits. If you insist on using them, make them a relatively inconspicuous feature of your reputation mechanism.

The story of Digg.com provides an example of the dangers of introducing too much competition in a community whose culture is otherwise meant to be cooperative. Digg.com is a system that allows users to recommend news articles that piqued their interest. Other users vote recommended articles up or down. An article’s number of votes determines its relative visibility on the site. The site was meant to be a grassroots method of determining “what is interesting to read about today.” Digg.com implemented a reputation mechanism that originally included a leaderboard of “most influential Diggers,” that is, of Digg members whose recommended articles received the most votes. Some of the leading Digg members became so obsessed with their rankings that they formed a collusion ring for the purpose of ensuring that their recommendations always made it to the top. This ended up eroding the culture and dynamics of the site. Amid controversy, in January 2007 Digg decided to abolish the Digg leaderboard. Today, Digg’s user profile consists of various “simple statistics” only.

Another important design dimension is whether a system bases its reporting on a user’s entire history on the site or whether statistics, scores or rankings are based on recent behavior only. Cumulative metrics keep increasing the longer a user stays on a site. That’s good for building loyalty but not so good for incentivizing continued contributions of seasoned users. It might also discourage new members from joining since they may judge it difficult or impossible to “catch up” with veteran members. Recent behavior metrics have the opposite properties: they keep members on their toes, they give newcomers equal opportunities but they reduce the cost of moving to a competitor’s site. A judicious combination of both types of metrics usually works best.

After several iterations, eBay and Amazon have both converged on such combinations. eBay’s reputation profile now includes both a feedback score, which is a cumulative metric, as well as a breakdown of a member’s recent feedback. Amazon now publishes two different rankings of reviewers: one primarily based on cumulative contributions and another based on recent contributions.