Worsening file-drawer problem in the abstracts of natural, medical and social science databases

Abstract

The file-drawer problem is the tendency of journals to preferentially publish studies with statistically significant results. The problem is an old one and has been documented in various fields, but to my best knowledge there has not been attention to how the issue is developing in a quantitative way through time. In the abstracts of various major scholarly databases (Science and Social Science Citation Index (1991)(1992)(1993)(1994)(1995)(1996)(1997)(1998)(1999)(2000)(2001)(2002)(2003)(2004)(2005)(2006)(2007)(2008), CAB Abstracts and Medline (1970s-2008), the file drawer problem is gradually getting worse, in spite of an increase in (1) the total number of publications and (2) the proportion of publications reporting both the presence and the absence of significant differences. The trend is confirmed for particular natural science topics such as biology, energy and environment but not for papers retrieved with the keywords biodiversity, chemistry, computer, engineering, genetics, psychology and quantum (physics). A worsening file-drawer problem can be detected in various medical fields (infection, immunology, malaria, obesity, oncology and pharmacology), but not for papers indexed with strings such as AIDS/HIV, epidemiology, health and neurology. An increase in the selective publication of some results against some others is worrying because it can lead to enhanced bias in metaanalysis and hence to a distorted picture of the evidence for or against a certain hypothesis. Long-term monitoring of the file-drawer problem is needed to ensure a sustainable and reliable production of (peer-reviewed) scientific knowledge.

Publication Growth in Biological Sub-Fields: Patterns, Predictability and Sustainability

“…Although the number of biological publications is increasing year after year, there is no evidence that publishing peer-reviewed papers is becoming any easier, as suggested by the report of a worsening file-drawer problem in natural, medical and social science databases [26]. A worsening publication bias towards positive results is a worrying trend because it may be a misleading factor in meta-analyses.…”

Meta-assessment of bias in science

“…The ratio of studies concluding in favor vs. against a tested hypothesis increases, moving from the physical, to the biological and to the social sciences, suggesting that research fields with higher noise-to-signal ratio and lower methodological consensus might be more exposed to positive-outcome bias (5,8,9). Furthermore, multiple independent studies suggested that this ratio is increasing (i.e., positive results have become more prevalent), again with differences between research areas (9)(10)(11), and that it may be higher among studies from the United States, possibly due to excessive "productivity" expectations imposed on researchers by the tenure-track system (12)(13)(14). Most of these results, however, are derived from varying, indirect proxies of positive-outcome bias that may or may not correspond to actual distortions of the literature.…”

New Opportunities

“…Studies involving pre-registered randomized trials that do not support study hypotheses are well suited to this mechanism. Advancing Science articles are intended to address a problem in behavioral and social science research known as the "file-drawer problem"-the failure to include null findings from rigorous research into empirical reviews and meta-analyses, and as benchmarks for estimating power and the potential impact of interventions and policies (Ioannidis, 2014;Pautasso, 2010;Rosenthal, 1979). Select publication of such research advances science by helping scholars establish more accurate effect size benchmarks when designing risk or protective factor studies, planning intervention research, or conducting reviews and meta-analyses.…”