Measuring Culture in Leading Companies

Introducing the MIT SMR/Glassdoor Culture 500

by: Donald Sull, Charles Sull, and Andrew Chamberlain

Culture Matters for Corporate Performance

On Feb. 5, 2018, Wells Fargo, then the third largest bank in the United States by assets, lost nearly $30 billion in market capitalization in a single day. The Federal Reserve had barred the bank from growing, as a penalty for opening more than 2 million accounts without authorization from customers.1 An investigation commissioned by the bank’s independent directors concluded that the root cause of the fraudulent behavior was the culture of the division whose employees opened the accounts.2

Culture matters. Now we can measure it.


Culture matters

See how major companies in the world economy rank across the nine dimensions of corporate culture, based on more than one million Glassdoor reviews.


Explore the Culture 500 »

The link between a toxic corporate culture and unethical behavior would come as no surprise to most executives. According to a recent survey, 85% of CEOs and CFOs believe that an unhealthy corporate culture leads to unethical behavior.3 The cost of a dysfunctional culture can be substantial. Public companies caught committing corporate fraud lose, on average, 25% to 44% of the value of their equity.4

On the flip side, a healthy company culture can turbocharge corporate performance. The same survey of CEOs and CFOs found that 9 out of 10 believe that improving corporate culture would increase their company’s value, and nearly 80% ranked culture among the five most important factors driving their company’s valuation. A growing body of research by financial economists has shown that a good corporate culture is correlated with higher profitability and returns to shareholders.5 Companies listed among the best places to work based on their corporate culture, for example, delivered nearly 20% higher returns to shareholders than comparable companies over a five-year period.6

Measuring Culture With Glassdoor Data

Culture matters for performance, but how can we quantify corporate culture and compare it across organizations? To understand how a company’s culture plays out in practice, we would need candid data from a large cross section of employees. To dig beneath simplistic assessments of culture as good or bad requires nuanced descriptions of the specific elements of corporate culture that are working well or poorly. Consistent data across a large number of companies is required to benchmark corporate cultures. Finally, linking culture to results requires data that can reliably predict corporate outcomes, financial performance, or innovation.

As it turns out, such a data source exists, and you may have used it when researching potential employers. Glassdoor is one of the largest job and recruiting sites in the world. Since its launch in 2008, Glassdoor has collected more than 49 million reviews and employee insights, covering approximately 900,000 organizations. Employees submit anonymous reviews, which means they can offer their candid opinions without fear of reprisal. Companies, moreover, cannot remove critical reviews. By aggregating these reviews, we can construct a comprehensive picture of a company’s culture that moves beyond the anecdotes and personal observations that managers often rely on to understand their corporate culture.

On Glassdoor, employees rate their company’s culture and values on a five-point scale, but these quantitative scores alone shed little insight on the specifics of a company’s culture. The real value for understanding and measuring culture lies in the free text responses that each reviewer provides. Here, employees describe — in their own words — the pros and cons of working at a particular company and offer advice to management. By analyzing this textual data, we can assess how well a company is doing on critical dimensions of culture — including diversity, collaboration, or integrity — in the eyes of employees.

Glassdoor reviews also provide clues to a company’s future performance. Wells Fargo’s reputation plummeted after regulators announced the bank’s financial fraud, but Glassdoor reviews signaled the bank had a problem with corporate ethics well before the fraud was made public.7 A series of studies have used Glassdoor data to predict a range of corporate outcomes, including future profitability, stock market returns, innovation, customer satisfaction, and financial fraud.8 In the years prior to the scandal, Wells Fargo employees were nearly twice as likely to discuss integrity in their reviews, and half as likely to discuss the bank’s ethics in positive terms compared with other large banks.9

To create the MIT SMR/Glassdoor Culture 500, we analyzed 1.2 million reviews using a natural language processing (NLP) methodology that accurately classifies free text into more than 90 culture-related topics. The combination of Glassdoor’s rich data set and our NLP algorithms allows us to present, for the first time, an online tool to compare the corporate cultures across companies that collectively employ 34 million people — the equivalent of one-quarter of private sector employment in the United States.10 This article describes how we define culture, the advantages of the Glassdoor reviews, how we analyzed the free text, and how to use the Culture 500 interactive tool to see how companies’ cultures stack up.

Defining Culture as Values and Norms

Most large companies describe their official corporate culture in terms of the values they want employees to practice on a day-to-day basis. Over 75% of the Culture 500 companies published an official statement of their corporate culture listing the specific values, such as integrity, customer-centricity, or respect, that together defined the culture the company aspired to achieve.11 Of course, employees do not always live up to their company’s espoused values, but official statements nevertheless shed light on which elements of corporate culture executives consider most important.

For the Culture 500, we follow Charles O’Reilly and Jennifer Chatman, who define organizational culture as “a set of norms and values that are widely shared and strongly held throughout the organization.”12 Shared values express what is important to employees, while norms provide clear guidance on behavior that is consistent with those values. Employees might all agree, for example, that integrity is worthwhile. But an organization needs norms — such as employees speak up when they see ethical breaches and managers protect whistle-blowers — to translate an abstract value like integrity into concrete behavior. This definition has been widely adopted by researchers studying the relationship between corporate culture and performance.13

Identifying the Big Nine Cultural Values

Given the diversity of corporate values across different companies, we faced the practical question of addressing which values matter most. In a separate study, we identified more than 60 distinct values that companies listed in their corporate values statements.14 Measuring and reporting dozens of values would be overwhelming, so we narrowed down the values to the nine that were cited most frequently by companies.15 These values, which we call the Big Nine, are agility, collaboration, customer, diversity, execution, innovation, integrity, performance, and respect.

Big Nine Cultural Values

For each of the Big Nine, we calculate the percentage of each company’s Glassdoor reviews that mention the value (incidence) as well as the percentage of reviews that discuss the value in positive terms (sentiment). The Culture 500 interactive tool provides users a snapshot of how frequently and positively employees within a company talk about the Big Nine values.

To illustrate culture at a glance within a company, consider the example of Amazon. (See “What Culture Looks Like at Amazon.”)

What Culture Looks Like at Amazon

This visualization shows how employees at Amazon rate culture across the Big Nine values in Glassdoor reviews. The values are normalized so that 0 is the average score for each Big Nine value for the company, and the axis labels denote standard deviations above or below the average when it comes to frequency (how often employees discuss this value) and sentiment (how positively employees discuss this value).

Consistent with the company’s official culture, which emphasizes customer obsession and innovation, we see that Amazon employees were most positive about innovation and assessed the retailer’s customer orientation very favorably as well. On the negative side, Amazon employees were much less enthusiastic about how well the company respects its workers.

Internally, Amazon employees rate the company highly on innovation, but how does Amazon stack up to other large information technology companies when it comes to innovation? The Culture 500 allows users to benchmark companies against their competitors on each of the Big Nine values. (See “How Tech Giants Compare on Innovation.”)

How Tech Giants Compare on Innovation

This visualization shows how companies considered giants in the tech industry measure up on innovation in Glassdoor reviews. The values are normalized so that 0 is the average score for innovation for all tech giants, and the axis labels denote standard deviations above or below this industry average when it comes to frequency (how often employees discuss this value) and sentiment (how positively employees discuss this value).

This industry comparison suggests that Amazon excels at innovation, even when benchmarked against other tech giants.

With the Culture 500 interactive tool, users can also construct their own customized set of companies and compare them along any of the Big Nine values. For instance, we can look at how Amazon measures up against companies from other industries — Nordstrom, Disney Parks, and the Four Seasons Hotels and Resorts — that enjoy excellent reputations for customer centricity. (See “Comparing Customer-Centricity Across Different Companies.”)

Comparing Customer-Centricity Across Different Companies

The Culture 500 allows direct company comparisons across industries. Here, we see how customer focus at Amazon compares with other customer-obsessed companies from other industries. The numbers denote percentiles across all companies in the Culture 500 sample.

Using Glassdoor Data to Understand Corporate Culture

Glassdoor is a gold mine of information for job seekers, and the Culture 500 demonstrates how employee reviews can help leaders understand their organization’s culture as well. In their open text responses, employees are free to talk about whatever matters most to them. Nearly all employees — 93% by our analysis — discuss some element of culture in their reviews, and 41% discuss one or more of the Big Nine values.16

Any single review could reflect an employee’s idiosyncratic view, but when hundreds or thousands of reviews are aggregated together, they paint an accurate picture of a company’s culture. The average company in our sample had 2,182 reviews, which represent about 4% of its total employment.17 To put this number in perspective, consider that the typical political poll surveys a sample of 1,000 people — approximately 0.001% of all registered voters in the United States — to assess the political climate.18

Understanding Spikes in Glassdoor Reviews

A large sample of reviews means nothing unless they accurately represent the views of a company’s workforce as a whole. A common concern with online reviews is that they skew toward a polarized distribution with many glowing reviews, a disproportionate number of extremely negative ones, and relatively few moderate opinions. The polarization of online reviews is well documented in a range of settings, from products on Amazon to restaurant reviews on Yelp.19 This polarized pattern occurs when only those users with extremely positive or extremely negative views are motivated to write a review.20

Yelp Versus Glassdoor Reviews

Like many online ratings platforms, Yelp reviews tend toward the extremes of very positive or very negative. Glassdoor minimizes the polarization through a series of user policies, creating a more equal and representative distribution of user ratings.

Unlike Yelp, Glassdoor reviews gravitate toward the center of the distribution with fewer extremely positive or negative ratings. (See “Yelp Versus Glassdoor Reviews.”)21 A recent study compared the polarization of reviews across online rating platforms based on the percentage of extreme reviews.22 Of the 25 online platforms analyzed, 21 were more polarized than Glassdoor.

Glassdoor minimizes polarization through a series of policies designed to promote honest and representative reviews. Employees, for example, can post no more than one review per company per year. This prevents vocal workers from stuffing the ballot box. Glassdoor has also minimized polarization with its “give to get” policy, which requires users to leave a review in order to access content on the website.

Reviews might also not present an accurate picture of organizational culture if companies incentivize or pressure employees to write glowing evaluations. Glassdoor has explicit community standards that prohibit companies from coercing, coaching, or incentivizing positive reviews. The company uses algorithms to detect content that might violate community standards, and users can flag questionable content. Glassdoor moderators review all flagged content and reject approximately 10% of submitted reviews deemed to violate community standards. There are also consequences for companies that try to cheat — employers caught violating community standards can be banned from consideration for Glassdoor’s Best Places to Work list and other awards.

We would expect to see large spikes in the number of positive reviews if companies were trying to incentivize reviews, but not all spikes indicate a company is gaming the system. Several factors can account for a surge in the number of reviews a company receives in a month. Some spikes occur purely by chance.23 News events — positive or negative — can also cause spikes. Intel, for example, experienced an upsurge in the number of reviews after announcing it would exit the smartphone chip market and lay off 12,000 employees.24 Companies can also ask employees to write honest reviews without attempting to influence the content. Increasing the number of reviews without skewing their distribution is good news since it provides us with more data to analyze.

To detect potentially problematic reviews, we identified months for companies that saw two or more standard deviations above the monthly mean for the number of reviews and the overall rating for the company, which we call positive spikes. Positive spikes were exceedingly rare — accounting for 0.1% of the company months in our sample. Nor do all positive spike months result from companies violating community standards. Some positive spikes happen by chance and others could be triggered by positive news. To be on the safe side, however, we dropped all positive spike months from our sample. When the positive spikes were removed, the remaining months with a large number of reviews were indistinguishable from our full sample.25

Codifying Culture With Machine Learning and Human Expertise

When describing culture, Glassdoor reviewers deploy rich and varied language — full of slang, idioms, acronyms, buzzwords, and colorful phrases. Nearly 80,000 employees discussed integrity in their reviews, for example, but less than 5,000 mentioned that specific word. Instead, employees used hundreds of synonyms and antonyms (ethics, deceit, amoral), idioms (sweep under the rug, cut corners), and slang (shady, shenanigans). Their language paints a nuanced picture of corporate culture but also defies simplistic approaches to textual analysis.

We initially analyzed the Glassdoor data with standard natural language processing tools, but the results were disappointing.26 To accurately classify text into topics that matter to managers and researchers, we built a custom dictionary from the ground up by combining machine learning tools and human expertise. This approach dramatically improved the accuracy of our models.27

To create a customized machine learning dictionary, we began by selecting topics that companies frequently included in their corporate values or that had been researched extensively. We then constructed a preliminary dictionary of high-fidelity words or phrases for each topic, and two of our research team members analyzed a random sample of at least 20 reviews that included each term. Terms that were at least 90% accurate were included in the dictionary for a topic. Ethical, for example, was a first ballot shoo-in for the integrity dictionary. If a word or phrase was less than 30% accurate, we dropped it. Terms that fell in between were fine-tuned until they could be included in the dictionary or dropped. This bootstrapping approach produced tens of thousands of terms, which human experts reviewed and refined.

To improve the accuracy of our results, we specified the sequence of words in a phrase — backdating has a very different meaning than dating back — as well as the distance between the words. Rather than applying a one-size-fits-all approach to handling close variants of a word, we differentiated between different uses. Employees, for example, typically use the word management to describe top leaders, manager when talking about their own boss, and manage to describe, for example, how they manage to get their work done despite distractions. We also identified words that meant a reviewer was not talking about a specific topic. The word vision typically refers to a company’s strategy, unless it appears in a sentence with dental, pension, or 401(k), in which case it describes a company’s benefits package.

For each term, we specified its component words, close variants, sequence and distance between words in a phrase, exclusion terms, and common misspellings. We also incorporated company-specific language. In most companies, for example, a shopper is a customer, but in Instacart a shopper is a worker. We reviewed the refined terms in context and also tested their fit with a topic by using a variety of algorithmic tools. This painstaking process took three years and produced a dictionary consisting of over 20,000 highly accurate terms mapping to more than 90 topics.

Exploring the Culture 500 Companies

The average company in our sample has over 2,000 employee reviews — the equivalent of three full-length books’ worth of textual data.28 Nearly all of the organizations in the Culture 500 are for-profit companies, and most are headquartered in the U.S. (We have included some international companies with large numbers of employees based in the U.S.) Regardless of where the company is headquartered, we included reviews only from employees based in the U.S.

Companies in the Culture 500 are divided into 33 clearly defined industries, with an average of 18 companies per industry.29 (See “Overview of Culture 500 Industries.”) Organizing companies into industries with at least 10 competitors allows for meaningful comparisons of corporate culture. It is not particularly useful to compare Amazon to Five Guys Burgers or Tim Hortons on innovation, but benchmarking Amazon against Facebook, Alphabet, Apple, and Netflix provides real insight. Companies that compete in multiple industries — Johnson & Johnson, for example, makes medical devices, drugs, and consumer goods — are included in more than one industry. We also include lists that cut across industries, like companies with the most patents and the most-admired companies.30

Overview of Culture 500 Industries

The Culture 500 represents 33 industries with an average of 18 companies within each industry. The number to the right of each bar graph denotes the number of companies included in that industry.

How Leaders Can Use the Culture 500

The Culture 500 provides a data-based view of the corporate cultures of some of the largest and most powerful organizations in the world. Leaders of these companies can benchmark their organizations against direct competitors and also best-in-class corporations that excel, for example, in the kind of agility, innovation, and customer orientation required to thrive in dynamic markets. Corporate leaders can also gauge whether employees believe they are living up to their stated values by, for instance, respecting every worker, acting with integrity, and providing an inclusive environment for a diverse workforce. Armed with these insights, executives can highlight cultural strengths to incorporate into their employer brand, spot warning signals for potential problems, and identify opportunities to improve their culture. They can also analyze acquisition and partnership candidates to identify cultural mismatches that could derail deals.

Entrepreneurs, likewise, can use the Culture 500 to assess the cultural fit between their startup and a large corporation before entering into a partnership or being acquired. They can identify companies or industries that struggle with innovation or customer orientation that could be targets for disruption. (See “Innovation Sentiment by Industry in the Culture 500,” which ranks industries in terms of employees’ assessment of innovation. It shows how employee reviews can predict industry changes, such as fintech startups’ disruption of financial services — including regional banks, insurance, and investment services industries.)

Innovation Sentiment by Industry in the Culture 500

This chart shows which industry’s employees rate innovation most favorably within Glassdoor reviews. This chart is based on over 1.2 million reviews. The numbers denote the number of standard deviations above or below the mean sentiment score across all industries.

Corporate culture matters beyond its power to drive financial results. The average employee spends 40% of her waking hours at work. A vibrant culture can help people thrive professionally, enjoy their job, and find meaning in their work. A toxic culture, in contrast, can be soul destroying. Large corporations are arguably the most powerful players in the global economy. By committing to values like diversity, integrity, and respect, influential companies can raise the bar of what is possible and desirable for competitors, suppliers, partners, and startups. Organizations that compromise their values breed cynicism and undermine the norms required for society as a whole to flourish. The Culture 500 celebrates excellence in culture, rewarding it with public recognition, and spurring it forward by identifying areas where some of the world’s most influential companies can improve their culture.

About the Partners

MIT Sloan Management Review

MIT Sloan Management Review leads the discourse among academic researchers, business executives, and other influential thought leaders about advances in management practice that are transforming how people lead and innovate. MIT SMR disseminates new management research and innovative ideas so that thoughtful executives can capitalize on the opportunities generated by rapid organizational, technological, and societal change.

Glassdoor

Glassdoor combines all the latest jobs with millions of reviews and insights to make it easy for people to find a job that is uniquely right for them. The company is on a mission to help people everywhere find a job and company they love. In pursuit of the mission, Glassdoor helps employers hire truly informed candidates at scale through effective recruiting solutions like job advertising and employer branding products. Launched in 2008, Glassdoor now has reviews and insights for approximately 900,000 companies located in more than 190 countries.

CultureX

CultureX delivers actionable insights organizations need to measurably improve their cultures. Harnessing cutting-edge artificial intelligence developed at MIT, CultureX measures culture with high accuracy and pinpoints concrete ways to improve. Based on decades of research and work with dozens of Fortune Global 500 companies, CultureX provides evidence-based interventions tailored to the client’s unique needs. CultureX builds vibrant cultures to enrich the lives of employees and win in the marketplace.

About the Authors

Donald Sull is a senior lecturer at the MIT Sloan School of Management and cofounder of CultureX. Charles Sull is a cofounder of CultureX. Andrew Chamberlain is chief economist at Glassdoor.

Contributors

Desiree Barry, Sean Brown, Carolyn Ann Geason, Ryan Melehan, Allyson MacDonald, Lauren Rosano, Pedro Henrique Santos, James Yoder

References

1. E. Glazer, “Fed Rebuke Costs Wells Fargo About $29 Billion in Lost Market Value,” The Wall Street Journal, Feb. 5, 2018.

2. Wells Fargo, “Independent Directors of the Board of Wells Fargo & Company Sales Practices Investigation Report,” April 10, 2017.

3. J.R. Graham, C.R. Harvey, J. Popadek, et al., “Corporate Culture: Evidence From the Field,” working paper 23255, National Bureau of Economic Research, Cambridge, Massachusetts, March 2017.

4. A. Dyck, A. Morse, and L. Zingales, “How Pervasive Is Corporate Fraud?” working paper, April 2017.

5. A. Edmans, “Does the Stock Market Fully Value Intangibles? Employee Satisfaction and Equity Prices,” Journal of Financial Economics 101, no. 3 (September 2011): 621-640; L. Guiso, P. Sapienza, and L. Zingales, “The Value of Corporate Culture,” Journal of Financial Economics 117, no. 1 (July 2015): 60-76; E. Symitsi, P. Stamolampros, and G. Daskalakis, “Employees’ Online Reviews and Equity Prices,” Economics Letters 162, issue C (2018): 53-55; T.C. Green, R. Huang, Q. Wen, et al., “Crowdsourced Employer Reviews and Stock Returns,” Journal of Financial Economics, forthcoming; and A. Chamberlain, “Does Company Culture Pay Off? Analyzing Stock Performance of ‘Best Places to Work’ Companies,” Glassdoor research report, Mill Valley, California, March 2015.

6. Edmans, “Does the Stock Market Fully Value Intangibles?”

7. The Harris Poll, “Corporate Reputation Politically Polarized as Companies Wrestle With Taking a Stand for Their Values,” PR Newswire, Feb. 9, 2017, www.prnewswire.com.

8. For articles using Glassdoor data to predict financial returns, see Green et al., “Crowdsourced Employer Reviews and Stock Returns” and Chamberlain, “Does Company Culture Pay Off? Analyzing Stock Performance of ‘Best Places to Work’ Companies”; M. Huang, P. Li, F. Meschke, et al., “Family Firms, Employee Satisfaction, and Corporate Performance,” Journal of Corporate Finance 34 (October 2015): 108-127; J. Hales, J.R. Moon Jr., and L.A. Swenson, “A New Era of Voluntary Disclosure? Empirical Evidence on How Employee Postings on Social Media Relate to Future Corporate Disclosures,” Accounting, Organizations, and Society 68-69 (July 2018): 88-108. For research using Glassdoor data to predict financial fraud, see Y. Ji, O. Rozenbaum, and K.T. Welch, “Corporate Culture and Financial Reporting Risk: Looking Through the Glassdoor,” working paper, SSRN, June 1, 2017; M. Huang, A. Masli, F. Meschke, et al., “Clients’ Workplace Environment and Corporate Audits,” Auditing: A Journal of Practice & Theory 36, no. 4 (November 2017): 89-113. For research on corporate culture and innovation, see M. Corritore, A. Goldberg, and S.B. Srivastava, “Duality in Diversity: How Intrapersonal and Interpersonal Heterogeneity Relate to Firm Performance,” Administrative Science Quarterly, April 17, 2019; K-T. Nguyen, “Trust and Innovation Within the Firm: Evidence From Matched CEO-Firm Data,” working paper, London School of Economics, January 2019. For research on culture and customer satisfaction, see J.S. Wolter, D. Bock, J. Mackey, et al., “Employee Satisfaction Trajectories and Their Effect on Customer Satisfaction and Repatronage Intentions,” Journal of the Academy of Marketing Science, May 18, 2019.

9. On Sept. 8, 2016, the Consumer Financial Protection Bureau (CFPB) announced Wells Fargo would pay $185 million in fines. In the two years prior to the CFPB’s action, Wells Fargo employees mentioned terms related to integrity, honesty, and ethics 1.8 times more frequently than employees in other large, diversified financial service firms in our sample over the same time period. When ethical issues were raised, Wells Fargo employees discussed them in positive terms at less than half the frequency (48%) than employees in other diversified financial institutions expressed positive sentiment about their bank’s ethics.

10. According to the U.S. Bureau of Labor Statistics, approximately 128 million people were employed in the private sector in the United States in April 2019.

11. D. Sull, S. Turconi, and C. Sull, “When It Comes to Corporate Culture, Does Your Company Walk the Talk?” working paper, MIT Sloan School of Management, Cambridge, Massachusetts. Guiso and his coauthors found that 85% of companies in the S&P 500 in 2011 listed their official values on their corporate website. See Guiso et al., “The Value of Corporate Culture”: 64.

12. C.A. O’Reilly and J.A. Chatman, “Culture as Social Control: Corporations, Cults, and Commitment,” in Research in Organizational Behavior, Vol. 18, eds. B.M. Staw and L.L. Cummings (Greenwich, Connecticut: JAI Press, 1996): 166.

13. See, for example, Guiso et al., “The Value of Corporate Culture”; L. Zingales, “The ‘Cultural Revolution’ in Finance,” Journal of Financial Economics 117, no. 1 (July 2015); Graham et al., “Corporate Culture”; Corritore et al., “Duality in Diversity”; J.A. Chatman, D.F. Caldwell, C.A. O’Reilly, et al., “Parsing Organizational Culture: How the Norm for Adaptability Influences the Relationship Between Culture Consensus and Financial Performance in High-Technology Firms,” Journal of Organizational Behavior 35, no. 6 (August 2014): 785-808.

14. The first two authors along with Stefano Turconi reviewed the websites of over 600 large companies (including all companies in the Culture 500). We identified 64 distinct values that were listed by at least 1% of the companies in our sample. For a list of the values and a description of our methodology, see Sull et al. “When It Comes to Corporate Culture.” The Organizational Culture Profile, a compendium of corporate values widely used by management scholars, also lists more than 60 values. See C.A. O’Reilly, J. Chatman, and D.F. Caldwell, “People and Organizational Culture: A Profile Comparison Approach to Assessing Person-Organization Fit,” Academy of Management Journal 34, no. 3 (1991): 487-516.

15. We began with a list of the 12 most commonly listed official values among companies in our sample. The three values we excluded from the Big Nine are people, excellence, and social responsibility. The term “people,” as commonly used in company descriptions and value statements (“We believe in people,” for example), was too vague to measure in the Glassdoor data. The term “excellence” similarly was used as a general term to refer to multiple organizational dimensions and values. Finally, social responsibility relates more to a company’s interactions with external stakeholders than with day-to-day behavior of employees throughout the organization, so it was excluded from the final list. We consolidated closely related values together for integrity (such as honesty), agility (speed and entrepreneurship), and execution (accountability). The Big Nine are broadly consistent with values identified by earlier research. All of the Big Nine were included among the values listed in the Organizational Culture Profile. Charles O’Reilly and his coauthors conducted a principal components analysis of the values listed in the OCP and identified six broad dimensions of culture, which they labeled adaptability, integrity, collaborative, results oriented, customer oriented, and detail oriented. See C.A. O’Reilly, D.F. Caldwell, J.A. Chatman, et al., “The Promise and Problems of Organizational Culture: CEO Personality, Culture, and Firm Performance,” Group & Organization Management 39, no. 6 (December 2014): 595-625. There is a one-to-one correspondence between four of their six dimensions and the Big Nine values (integrity, collaboration, results oriented/performance, and customer). The term “detail” only appeared twice in the descriptions of the 3,359 values we coded, and did not pass the frequency threshold required for inclusion as a separate value. We divided their dimension of adaptability into agility and innovation in the Big Nine. Diversity, respect, and execution were not included among the O’Reilly et al. cultural dimensions.

16. Nearly all (93%) of reviews mentioned at least one of the more than 90 topics that our algorithm codes. These represent a broad set of cultural topics, including values that are not included in the Big Nine (e.g., ambition, curiosity) as well as topics such as perks, work space, and benefits that are sometimes considered part of corporate culture even if they are not values.

17. Our analysis is based on reviews from Jan. 1, 2014, and March 31, 2019. Across our full sample of 531 organizations, the average number of reviews was 2,182 and the median number of reviews was 1,221. We compared the number of reviews to the number of employees two ways. First, we used Glassdoor’s estimated number of employees per employer. For the 531 companies in our sample, the average number of reviews per average number of employees was 3.4%, and the median number of reviews per median number of employees was 4.5% using the Glassdoor employee data. To validate these estimates, we took a subset of 327 publicly listed companies in our sample that listed their total employment in Securities and Exchange Commission filings. For this subset, the average number of reviews was 2,186, which represented 4.1% of the average number of employees for this subset of public companies. The median number of employees was 1,172, which represented 3.7% of the median number of employees in the subset of 327 companies.

18. For Gallup’s methodology and sample size, see “How Does the Gallup U.S. Poll Work?” www.gallup.com. For another reference point, Fortune magazine surveys approximately 5% of participating companies’ workforce to assess corporate culture when compiling its annual Best Companies to Work For list. See also: Guiso et al., “The Value of Corporate Culture”: 65.

19. For evidence of polarized online reviews, see J.A. Chevalier and D. Mayzlin, “The Effect of Word of Mouth on Sales: Online Book Reviews,” Journal of Marketing Research 43, no. 3 (Aug. 1, 2006): 345-354; N. Hu, P.A. Pavlou, and J. Zhang, “Why Do Online Product Reviews Have a J-Shaped Distribution? Overcoming Biases in Online Word-of-Mouth Communication,” Communications of the ACM 52, no. 10 (March 1, 2007): 144-147; M. Luca and G. Zervas, “Fake It ’Til You Make It: Reputation, Competition, and Yelp Review Fraud,” Management Science 62, no. 12 (December 2016): 3393-3672; S.F. Lu and H. Rui, “Can We Trust Online Physician Ratings? Evidence From Cardiac Surgeons in Florida,” Management Science 64, no. 6 (June 2018): 2473-2972.

20. For recent reviews of how self-selection of reviewers influences the distribution of online reviews, see X. Li and L.M. Hitt, “Self-Selection and Information Role of Online Product Reviews,” Information Systems Research 19, no. 4 (December 2008): 397-521. For an overview of factors that induce people to write online reviews, see W.W. Moe, D.A. Schweidel, and M. Trusov, “What Influences Customers’ Online Comments,” MIT Sloan Management Review 53, no. 1 (fall 2011): 14-16.

21. Yelp, “An Introduction to Yelp Metrics,” review of cumulative Yelp reviews across all categories through Dec. 31, 2018, www.yelp.com.

22. Verena Schoenmueller and her coauthors define polarity as the number of extreme reviews (1 and 5 on a five-point scale) divided by the total number of reviews, and measure polarity across 25 online review platforms. They find that polarity varies from 9% (RateBeer) to 93% (Frag Mutti). See V. Schoenmueller, O. Netzer, and F. Stahl, “The Extreme Distribution of Online Reviews: Prevalence, Drivers, and Implications,” Columbia Business School research paper no. 18-10, Feb. 15, 2019. Using the author’s measure of polarity, the distribution of overall ratings in the Culture 500 sample is 29% (19% five star + 10% one star), which we compared to the polarity scores for the 25 online rating platforms listed in Table 1 of the paper.

23. We tallied the number of reviews for each Culture 500 company on a monthly basis and identified those months when a company experienced a significant surge in reviews (two or more standard deviations above the average number of reviews that company received in a month). We called these spike months. There were 1,079 spike months, which represented 3.2% of all company months. If the volume of reviews were normally distributed, we would expect 2.3% of months to be spikes.

24. V. Savov, “Intel’s New Smartphone Strategy Is to Quit,” The Verge, May 3, 2016, www.theverge.com.

25. After we eliminated the 47 positive spike months from our sample, we compared the remaining 1,032 spike months to the full sample of all company months. The remaining spike months were 0.06 standard deviations above the average overall rating for all months in our sample. The percentage of five-star reviews was 0.04 standard deviations above the full sample mean.

26. We initially used latent Dirichlet allocation (LDA) to allocate terms to topics, but this approach produced only a few useful categories (e.g., benefits, perks, job security). The other topics generated by LDA were either too broad to be useful (management, processes) or lumped together terms that were conceptually and practically distinct from one another. The topics generated by this approach, moreover, did not map well to the values that companies listed as elements of their culture or to constructs that academics had identified as important for corporate performance. These algorithmically generated categories did a poor job of predicting classifications of text produced by expert human coders. To test the topics generated by LDA, we created a test set of 4,500 randomly chosen text instances (pros, cons, or advice to management fields from Glassdoor reviews) for the electrical equipment industry. The first author and another trained coder classified each text instance for 14 topics based on whether it discussed or did not discuss a given topic. The inter-rater reliability was 92%. We fit a variety of models, including random forest and logistic regression models, to predict the human coding using the topics generated by LDA as features, along with unigrams and bigrams. The average F-score across all topics was 0.63.

27. For the ground-up approach, we created a set of terms corresponding to the 14 topics, and then included these in features to classify 4,500 text instances from the electrical equipment industry. The average F-score using the hand-curated terms increased to 0.92 across all 14 topics. To test whether we had overfit on a single sector, we selected a different industry (insurance) and took a random sample of 1,200 text instances from Glassdoor reviews of insurers. A trained coder, who had not analyzed the electrical equipment data, classified the insurance text instances on the same 14 topics. We then used the same curated terms as features to predict the hand-coding of the 14 topics in the insurance industry data. The average F-score using a random forest model was 0.86 across all 14 topics.

28. The average company in our sample had 2,182 reviews from January 2014 through March 2019, and the average review was 69 words long, which equals 150,558 words per company, or the equivalent of just over three 50,000-word-long books. Focusing on employers with a large number of reviews helps ensure that even infrequently discussed values will have a sufficient number of reviews for robust results. Diversity, for example, is the least frequently mentioned value, and is discussed by 4% of employees in our sample. The low incidence multiplied by the large number of reviews still translates into nearly 90 employees talking about diversity in the average Culture 500 company, which prevents a few extreme reviews from skewing our results.

29. We primarily organized companies using the Global Industry Classification Standard (GICS), which does a better job than alternative classification schemes in predicting a variety of financial outcomes. See S. Bhojraj, C.M.C. Lee, and D.K. Oler, “What’s My Line? A Comparison of Industry Classification Schemes for Capital Market Research,” Journal of Accounting Research 41, no. 5 (Nov. 14, 2003): 745-774. Sixteen of the 33 Culture 500 industries correspond most closely to GICS subindustries, 12 correspond to GICS industries, and 5 correspond to GICS industry groups. To increase the number of companies per industry, we expanded our sample of companies to 531 in total.

30. Top 30 U.S. companies in Fortune’s 2019 “The World’s Most Admired Companies” list. Mercedes Delgado of MIT Sloan generously shared her list of 30 companies with the most U.S. patents assigned between 2000-2015 (includes those assigned to subsidiaries).

i. I.E. Marinescu, N. Klein, A. Chamberlain, et al., “Incentives Can Reduce Bias in Online Reviews,” working paper, SSRN, Feb. 26, 2018.

Tags:

Reprint #:

61130

More Like This

Add a comment

You must to post a comment.

First time here? Sign up for a free account: Comment on articles and get access to many more articles.

Comments (4)
Gaurav Pandit
Louis Wang: How are you able to see Sentiment and Frequency scores for Ultimate Software and Danaher? Is that part of the report? Can only see Std. Deviation scores on the webpage.
Also, Danaher is not part of the list either. No.8 in Execution is SpaceX
Louis Wong
It seems that there are no correlation among these 9 Culture Values. What I mean is that  For example, Is better Agility leading to better rating in Execution?

Or these 9 Culture Values are totally independent.

Thanks,
Louis Wong
Louis Wong
I have troubles to understand the company ranking and it relationship with its own score in the culture values. 
For example, Ultimate Software ranked Number 4 in Execution with a standard Deviation of +3.7 her own scores in Execution are frequency +1.4 and Sentiment +/- 0.

However, Danaher ranked only No. 8 in Execution with a Standard Deviation of +3.1 but her own scores in Execution are frequency +2.4 and Sentiment is +1.3
I could not understand these results.

Would the team explain, please?

Thanks,
Louis Wong
Rebecca Casey
I am currently in OD nerd heaven.