An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017)

Abstract

Serious concerns about research quality have catalysed a number of reform initiatives intended to improve transparency and reproducibility and thus facilitate self-correction, increase efficiency and enhance research credibility. Meta-research has evaluated the merits of some individual initiatives; however, this may not capture broader trends reflecting the cumulative contribution of these efforts. In this study, we manually examined a random sample of 250 articles in order to estimate the prevalence of a range of transparency and reproducibility-related indicators in the social sciences literature published between 2014 and 2017. Few articles indicated availability of materials (16/151, 11% [95% confidence interval, 7% to 16%]), protocols (0/156, 0% [0% to 1%]), raw data (11/156, 7% [2% to 13%]) or analysis scripts (2/156, 1% [0% to 3%]), and no studies were pre-registered (0/156, 0% [0% to 1%]). Some articles explicitly disclosed funding sources (or lack of; 74/236, 31% [25% to 37%]) and some declared no conflicts of interest (36/236, 15% [11% to 20%]). Replication studies were rare (2/156, 1% [0% to 3%]). Few studies were included in evidence synthesis via systematic review (17/151, 11% [7% to 16%]) or meta-analysis (2/151, 1% [0% to 3%]). Less than half the articles were publicly available (101/250, 40% [34% to 47%]). Minimal adoption of transparency and reproducibility-related research practices could be undermining the credibility and efficiency of social science research. The present study establishes a baseline that can be revisited in the future to assess progress.

Estimating the prevalence of transparency and reproducibility-related research practices in psychology (2014-2017)

“…Replication or evidence synthesis via meta-analysis or systematic review was relatively infrequent. Although there is evidence that some recent methodological reform initiatives have been effective (Hardwicke et al, 2019;Nelson et al, 2018;Vazire, 2018), the findings of the current study imply that their impact on the psychology literature during the examined period was fairly limited in scope.…”

“…version ("open access"). This is higher than recent open access estimates obtained for biomedicine (25%; and the social sciences (40%; Hardwicke et al, 2019), as well as a large-scale automated analysis which suggested that 45% of the scientific literature published in 2015 was publicly available (Piwowar et al, 2018). Limiting access to academic publications reduces opportunities for researchers, policy-makers, practitioners, and the general public, to evaluate and make use of scientific evidence.…”

“…Meta-analysis citation history* (has the article been cited by, and included in the evidencesynthesis component of, a meta-analysis) Study designs involving primary data # Systematic review citation history* (has the article been cited by, and included in the evidencesynthesis component of, a systematic review) Study designs involving primary data # social sciences (Hardwicke et al, 2019). As indicated in the second column of the table, the exact variables measured for a given article depended on study design.…”

“…Close scrutiny of psychology in particular has suggested that standard research and publication practices have rendered the discipline highly exposed to bias, potentially resulting in a large volume of exaggerated and misleading results John et al, 2012;Open Science Collaboration, 2015;Pashler & Wagenmakers, 2012;Simmons et al, 2011;Szucs & Ioannidis, 2017). This realization within the field has also led to a number of reform efforts (Hardwicke & Ioannidis, 2018a;Hardwicke et al, 2019;Nelson et al, 2018;Vazire, 2018), which have the potential to improve efficiency (Chalmers & Glasziou, 2009), facilitate self-correction (Ioannidis, 2012), and enhance credibility (Vazire, 2017).…”

“…However, the prevalence of transparency and reproducibility-related research practices across the psychology literature is largely unknown. Based on previous investigations in biomedicine (Iqbal et al, 2016; and the social sciences (Hardwicke et al, 2019), we manually examined a random sample of 250 articles to estimate the prevalence of several transparency and reproducibility-related indicators in psychology articles published between 2014-2017. The indicators were open access to published articles; availability of study materials, study protocols, raw data, and analysis scripts; pre-registration; disclosure of funding sources and conflicts of interest; conduct of replication studies; and cumulative synthesis of evidence in meta-analyses and systematic reviews.…”