Achieving Statistical Significance with Covariates and without Transparency

Abstract

How often do articles depend on suppression effects for their findings? How often do they disclose this fact? By suppression effects, we mean control-variable-induced increases in estimated effect sizes. Researchers generally scrutinize suppression effects as they want reassurance that researchers have a strong explanation for them, especially when the statistical significance of the key finding depends on them. In a re-analysis of observational studies from a leading journal, we find that over 30% of articles depend on suppression effects for statistical significance. Although increases in key effect estimates from including control variables are of course potentially justifiable, none of the articles justify or disclose them. These findings may point to a hole in the review process: journals are accepting articles that depend on suppression effects without readers, reviewers, or editors being made aware.

Pre-analysis Plans: A Stocktaking

“…In 44% of PAPs, the number of pre-specified control variables was judged to be unclear, making it nearly impossible to compare what was pre-registered with what is ultimately presented in the resulting paper. Lenz and Sahn (2019) show that the flexibility stemming from such imprecision provides wide scope for generating results that might not otherwise have reached traditional levels of statistical significance. 14 Further scope for fishing comes from imprecision in the empirical models that are pre-specified.…”

“…and fails the Olken test. 14 Lenz and Sahn (2019) find that 30-40% of observational studies report findings that depend on covariates to increase their effect sizes to the point where they cross the threshold of statistical significance, and that the authors of these studies almost never disclose that their results depend on the particular constellation of covariates they have chosen to include. 15 The simulations in Humphreys, De la Sierra and Van der Windt (2013) suggest that discretion over model selection is not a major source of fishing opportunities.…”

Moral Obstinacy in Political Negotiations

“…Each specification has advantages. The first estimates the effect of moral conviction without covariates and without requiring assumptions about how moral conviction relates to the other measures (Lenz & Sahn, 2017). The second specification tests whether moral conviction has effects over importance and relevance—two attitude measures with a long history in the literature.…”

Is research in social psychology politically biased? Systematic empirical tests and a forecasting survey to address the controversy

“…For instance, an article showing a lack of gender differences on a specific dimension may be more cited (and less easily refuted) than an article reporting a lack of gender discrimination in a particular also be more likely to use covariates to achieve significance (Lenz & Sahn, 2017), and scientists' political orientation may moderate their effect size estimates on politicized topics when many analysts use the same dataset to test the same hypothesis (Silberzahn et al, in press).…”