Mapping the discursive dimensions of the reproducibility crisis: A mixed methods analysis

Abstract

Addressing issues with the reproducibility of results is critical for scientific progress, but conflicting ideas about the sources of and solutions to irreproducibility are a barrier to change. Prior work has attempted to address this problem by creating analytical definitions of reproducibility. We take a novel empirical, mixed methods approach to understanding variation in reproducibility conversations, which yields a map of the discursive dimensions of these conversations. This analysis demonstrates that concerns about the incentive structure of science, the transparency of methods and data, and the need to reform academic publishing form the core of reproducibility discussions. We also identify three clusters of discussion that are distinct from the main group: one focused on reagents, another on statistical methods, and a final cluster focused the heterogeneity of the natural world. Although there are discursive differences between scientific and popular articles, there are no strong differences in how scientists and journalists write about the reproducibility crisis. Our findings show that conversations about reproducibility have a clear underlying structure, despite the broad scope and scale of the crisis. Our map demonstrates the value of using qualitative methods to identify the bounds and features of reproducibility discourse, and identifies distinct vocabularies and constituencies that reformers should engage with to promote change.

No time for that now! Qualitative changes in manuscript peer review during the Covid-19 pandemic

“…Lowering the bar on the number of (control) experiments conducted or data sets included can have multiple implications for the quality of the science conducted. For several decades now, multiple scholars have claimed science to be in a ‘reproducibility crisis’ ( Guttinger 2020 ; Nelson et al. 2020 ).…”

Mapping the discursive dimensions of the reproducibility crisis: A mixed methods analysis

“…Metascientists have been central players in discussions about reproducibility, and have been especially active in research related to scholarly communication and open science [ 67 ]. However, time series analysis shows that the thematic core identified here has been present even from the very early days of reproducibility discussions [ 68 ], suggesting that it emerged before or alongside metascience rather than as a result of the formation of the field. The cluster of themes identified here is also present in articles by authors who would be unlikely to self-identify as metascientists, such as Francis Collins and Lawrence Tabak’s early paper outlining the NIH’s plans for enhancing reproducibility [ 19 ].…”