Scientific apophenia in strategic management research: Significance tests & mistaken inference

Abstract

Research summary: This article uses distributional matching and posterior predictive checks to estimate the extent of false and inflated findings in empirical research on strategic management. Based on a sample of 300 papers in top outlets for research on strategic management, we estimate that if each study were repeated, 24–40 percent of significant coefficients would become insignificant at the five percent level. Our best guess is that for about half of these, the true coefficient is very close to 0. The remaining coefficients are likely directionally correct but inflated in magnitude. We offer several practical individual and field level suggestions for reducing scientific apophenia, that is, our tendency to find and publish evidence of order where none exists. Managerial summary: This article analyzes the degree to which statistics in research on strategic management provide meaningful evidence for decision-making. Based on a sample of 300 papers, we estimate that 24%–40% of reported results would probably not be confirmed if the study were repeated. Our best guess is that about half of the reported results are wrong (B∼ = 0) and the other half of results too weak to find repeatedly. We conclude that scientific apophenia—the tendency to find evidence of order where none exists—is a serious problem in the literature on strategic management. We recommend replication of empirical studies to insure that they provide evidence for guiding managers. We also provide guidance for avoiding scientific apophenia in empirical research. Copyright © 2015 John Wiley & Sons, Ltd.

The Critique of Empirical Social Science: New Policies atManagement and Organization Review

“…They found that average effect sizes were considerably smaller than originally reported. Closer to home, Goldfarb and King (2016) assessed a sample of 300 published studies. They estimated that 24-40% of the studies could not be replicated.…”

Science’s reproducibility and replicability crisis: International business is not immune

“…Despite its many flaws, null hypothesis significance testing (NHST) continues to be the choice of researchers in management and organization studies (Bettis, Ethiraj, Gambardella, Helfat, & Mitchell, 2016;Meyer et al, 2017). In NHST, the tenability of a null hypothesis (i.e., no effect or relation) is primarily judged based on the observed p value associated with the test of the hypothesis, and values smaller than 0.05 are often judged as providing sufficient evidence to reject it (Bettis et al, 2016;Goldfarb & King, 2016). Of the many problems associated with this interpretation of p values, the most pernicious is that it motivates researchers to engage in a practice called ''p-hacking'' and to report ''crippled'' p values (see below) (Aguinis, Werner, Abbott, Angert, Park, & Kohlhausen, 2010;Banks, Rogelberg et al, 2016).…”

“…For example, consider a researcher who interprets p = 0.0499 as sufficient evidence for rejecting the null hypothesis, and p = 0.0510 as evidence that the null hypothesis should be retained, and believes that journals are more likely to look favorably on rejected null hypotheses. This researcher will be highly motivated to ''p-hack,'' that is, find some way, such as using control variables or eliminating outliers, to reduce the p value below the 0.05 threshold (Aguinis et al, 2010, Goldfarb & King, 2016Starbuck, 2016;Waldman & Lilienfeld, 2016). Similarly, this researcher will be motivated to report p values using cutoffs (e.g., p \ 0.05), rather that report the actual p value (0.0510).…”

“…In classical hypothesis testing, results either are or are not significant; there is no such thing as ''marginally significant'' results. The examples regarding the use of control variables and outliers provided above, along with evidence from other fields, such as strategic management (Bettis et al, 2016;Goldfarb & King, 2016) and psychology (Bakker & Wicherts, 2011;Nuijten, Hartgerink, Assen, Epskamp, & Wicherts, 2015) suggest the existence of published articles in which researchers exercised their ''degrees of freedom'' to systematically manipulate the data to obtain a significant (i.e., p \ 0.05) result. Engaging in these practices increases systematic capitalization on chance and diminishes the probability that results will be reproducible and replicable.…”

A Call for Openness in Research Reporting: How to Turn Covert Practices Into Helpful Tools

“…Based on 300 articles in prominent strategic management journals, Goldfarb and King (2015) estimated conservatively that about 25-40% of the published claims of statistical significance are actually false. Such audits strongly suggest that researchers or editors do not publish studies that report null-findings (Kepes et al, 2012).…”