An Evaluation of Nephrology Literature for Transparency and Reproducibility Indicators: Cross-Sectional Review

Abstract

Introduction: Reproducibility is critical to diagnostic accuracy and treatment implementation. Concurrent with clinical reproducibility, research reproducibility establishes whether the use of identical study materials and methodologies in replication efforts permits researchers to arrive at similar results and conclusions. In this study, we address this gap by evaluating nephrology literature for common indicators of transparent and reproducible research. Methods: We searched the National Library of Medicine catalog to identify 36 MEDLINE-indexed, Englishlanguage nephrology journals. We randomly sampled 300 publications published between January 1, 2014, and December 31, 2018. Results: Our search yielded 28,835 publications, of which we randomly sampled 300 publications. Of the 300 publications, 152 (50.7%) were publicly available, whereas 143 (47.7%) were restricted through paywall and 5 (1.7%) were inaccessible. Of the remaining 295 publications, 123 were excluded because they lack empirical data necessary for reproducibility. Of the 172 publications with empirical data, 43 (25%) reported data availability statements and 4 (2.3%) analysis scripts. Of the 71 publications analyzed for preregistration and protocol availability, 0 (0.0%) provided links to a protocol and 8 (11.3%) were preregistered. Conclusion: Our study found that reproducible and transparent research practices are infrequently used by the nephrology research community. Greater efforts should be made by both funders and journals. In doing so, an open science culture may eventually become the norm rather than the exception.

How often do cancer researchers make their data and code available and what factors are associated with sharing?

“…The observation of the low availability of data and code in cancer research, as well as medical research more broadly, is not new. Rather our observations that 19% and 4% of cancer researchers declared data and code were publicly available are consistent with several previous studies reporting low, but increasing, declaration rates ranging between 3-24% and 0-2% respectively across a number of other medical fields between 2014 and 2018 [14][15][16][17][18][19][20][21][22][23][24], as well as more recent estimates [25][26][27].…”

“…Specifically, we noted that only one of the 59 articles that declared data was available complied with our FAIR assessment. This observation, depending on how availability for reuse is defined, is unfortunately consistent with this body of research which has reported 50-100% reductions in availability following interrogation of sharing statements [14][15][16][17][18][19][20][21][22][23][24]; with factors such as the lack of unique and permanent identifiers, meta-data and licensing terms being noted as major pitfalls [32,52]. Furthermore, while we also noted a strong relationship between mandatory data sharing policies and actual data availability, we unfortunately also observed similarly sub-optimal compliance with these policies too; a finding that has been noted by other studies both inside and outside of medicine [28,53,54].…”

“…In contrast to the growth of data sharing declarations over time, despite claims that code sharing is becoming increasingly normalised across many scientific fields [58], we note persistently low code sharing rates in oncology and medicine more broadly since 2014 [14][15][16][17][18][19][20][21][22][23][24][25][26]. Furthermore, none of the six studies in our sample that were subject to mandatory code sharing policies reported code to be available.…”

“…We note some strengths of our study. First, our sample size was 4-10 times bigger than the annual estimates of previous research evaluating data and code sharing rates in both oncology and medicine more broadly between 2014 and 2018 [14][15][16][17][18][19][20][21][22][23][24]. Random sampling of articles also allowed us to make inferences about sharing rates more broadly than studies focused on sharing rates for articles published in specific journals (e.g.…”

“…While significant progress has been made towards increasing the availability of the products of research (i.e. data, code and materials), previous research on the topic has reported low availability of data (0-6%) and code (0-2%) across many medical fields between 2014 and 2018 and beyond [14][15][16][17][18][19][20][21][22][23][24][25][26][27]. Similar observations have also been made in oncology, with one study by Walter and colleagues [21] observing that only 2% and 0% of 45 sampled cancer publications in 2014 shared sufficient data and code to computationally verify all reported findings; with this rate not increasing over the subsequent three year period.…”