Think Critically About the Wisdom of Experts
Some lessons about assessing the claims of people whose opinions seem unassailable.
Expert analysis informs the decisions we make as leaders and managers — and in our everyday lives. We can’t see red blood cells, but we trust scientists who say we have them and doctors who order blood tests to count them. We suspect that cognitive biases affect our choices, not because we have done the analysis ourselves, but because we believe social scientists who conduct experimental research. Much of our knowledge is ultimately garnered from the testimony of teachers, mentors, colleagues, and authors who write for publications like this one.
But we also live in a world where, almost daily, some expert’s previous certainty is discredited by new analysis. Diets once thought to be foolproof are ridiculed; management practices once decried are suddenly praised. So how should we treat the next piece of advice we get from a scholar or a consultant?
Philosophers of science, who study this issue, generally recommend that we simply trust what we hear from well-credentialed people who seem competent and sincere. But I think we can do better. We should always think critically about what we hear or read.
In my experience, “fresh eyes” often find errors that have eluded expert minds. We owe it to ourselves to handle each item of expertise the way we would a piece of fruit we’re about to buy — gauging how wholesome and ripe it is. Here are my thoughts on how to do that.
Get Updates on Transformative Leadership
Evidence-based resources that can help you lead your team more effectively, delivered to your inbox monthly.
Please enter a valid email address
Thank you for signing up
Dare to Doubt
In the second most popular TED talk of all time,1 social psychologist Amy Cuddy tells us that holding certain physical postures boosts our power hormones and makes us more courageous; however, attempts to replicate that result have failed.2 European governments chose to adopt austerity policies in part because esteemed Harvard economists Carmen Reinhart and Kenneth Rogoff told them that high debt levels cause a sudden drop in economic growth.3 Then a graduate student, Thomas Herndon, discovered that their claim was influenced by an Excel spreadsheet error.4
Experts fool themselves all the time, especially when a problem is messy and the analysis is difficult. The replication crisis — whereby scientific findings are increasingly being revealed as tough to reproduce — is plaguing psychology, economics, and medical research.
References
1. A. Cuddy, “Your Body Language May Shape Who You Are,” presentation at TEDGlobal 2012: Radical Openness, Edinburgh, United Kingdom, June 25-29, 2012.
2. M.W. Berger, “Power Poses Don’t Help and Could Potentially Backfire, Penn Study Shows,” Penn Today, Nov. 23, 2016.
3. C.M. Reinhart and K.S. Rogoff, “Growth in a Time of Debt,” American Economic Review 100, no. 2 (May 2010): 573-578.
4. K. Roose, “Meet the 28-Year-Old Grad Student Who Just Shook the Global Austerity Movement,” Daily Intelligencer, April 18, 2013.
5. S. Vasishth, “The Replication Crisis in Science,” The Wire, Dec. 29, 2017.
6. J.P.A. Ioannidis, “Why Most Published Research Findings Are False,” PLoS Medicine 2, no. 8 (Aug. 30, 2005).
7. J.B. De Long and K. Lang, “Are All Economic Hypotheses False?” Journal of Political Economy 100, no. 6 (December 1992): 1,257-1,272.
8. B. Goldfarb and A.A. King, “Scientific Apophenia in Strategic Management Research: Significance Tests and Mistaken Inference,” Strategic Management Journal 37, no. 1 (January 2016): 167-176.
9. J. Han, M. Kamber, and J. Pei, “Mining Frequent Patterns, Associations, and Correlations,” chap. 5 in “Data Mining: Concepts and Techniques,” 2nd ed. (San Francisco: Morgan Kaufmann, 2006): 227-283.
10. A. Gelman and E. Loken, “The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No ‘Fishing Expedition’ or ‘P-Hacking’ and the Research Hypothesis Was Posited Ahead of Time,” unpublished ms, Nov. 14, 2013.
11. D.F. Hultsch, C. Hertzog, B.J. Small, and R.A. Dixon, “Use It or Lose It: Engaged Lifestyle as a Buffer of Cognitive Decline in Aging?” Psychology and Aging 14, no. 2 (June 1999): 245-263.
12. J. Weuve, C. Proust-Lima, M.C. Power, A.L. Gross, S.M. Hofer, R. Thiébaut, G. Chêne, M.M. Glymour, C. Dufoil, and MELODEM Initiative, “Guidelines for Reporting Methodological Challenges and Evaluating Potential Bias in Dementia Research,” Alzheimer’s & Dementia 11, no. 9 (September 2015): 1,098-1,109.
13. R.P. Feynman, “Cargo Cult Science,” Engineering and Science (June 1974): 10-13.
14. K. Jung, S. Shavitt, M. Viswanathan, and J.M. Hilbe, “Female Hurricanes Are Deadlier Than Male Hurricanes,” Proceedings of the National Academy of Sciences 111, no. 24 (June 17, 2014): 8,782-8,787.
15. D. Malter, “Female Hurricanes Are Not Deadlier Than Male Hurricanes,” Proceedings of the National Academy of Sciences 111, no. 34 (Aug. 24, 2014): E3,496; and L.A. Bakkensen and W. Larson, “Population Matters When Modeling Hurricane Fatalities,” Proceedings of the National Academy of Sciences 111, no. 50 (Dec. 16, 2014): E5,331-5,332.
16. I. Savage, “Comparing the Fatality Risks in United States Transportation Across Modes and Over Time,” Research in Transportation Economics 43, no. 1 (July 1, 2013): 9-22.
17. J.R. Lott and D.B. Mustard, “Crime, Deterrence, and Right-to-Carry Concealed Handguns,” Journal of Legal Studies 26, no. 1 (January 1997): 1-68.
18. D.A. Black and D.S. Nagin, “Do Right-to-Carry Laws Deter Violent Crime?” Journal of Legal Studies 27, no. 1 (January 1998): 209-219; and I. Ayres and J.J. Donohue III, “Shooting Down the More Guns, Less Crime Hypothesis,” working paper 9,336, National Bureau of Economic Research, November 2002.
19. National Research Council, “Firearms and Violence: A Critical Review” (Washington, D.C.: National Academies Press, 2005).
20. S.N. Durlauf, S. Navarro, and D.A. Rivers, “Model Uncertainty and the Effect of Shall-Issue Right-to-Carry Laws on Crime,” European Economic Review 81 (January 2016): 32-67.
21. R. Goldman and S. Klepper, “Spinoffs and Clustering,” RAND Journal of Economics 47, no. 2 (summer 2016): 341-365.
22. R. Feynman, “The Pleasure of Finding Things Out: The Best Short Works of Richard P. Feynman” (New York: Basic Books, 2005).
Comment (1)
Paul H Aube