An evaluation of Nephrology Literature for Transparency and Reproducibility Indicators: Cross-sectional Review

Abstract

Background:Reproducibility is critical to diagnostic accuracy and treatment implementation. Concurrent with clinical reproducibility, research reproducibility establishes whether the use of identical study materials and methodologies in replication efforts permit researchers to arrive at similar results and conclusions. In this study, we address this gap by evaluating nephrology literature for common indicators of transparent and reproducible research. Methods:We searched the National Library of Medicine catalog to identify 36 MEDLINE-indexed, English language nephrology journals. We randomly sampled 300 publications published between January 1, 2014, and December 31, 2018. In a duplicated and blinded fashion, two investigators screened and extracted data from the 300 publications. Results:Our search yielded 28,835 publications, of which we randomly sampled 300 publications. Of the 300 publications, 152 (50.67%) were publicly available whereas 143 (47.67%) were restricted through paywall and 5 (1.67%) were inaccessible. Of the remaining 295 publications, 123 were excluded because they lack empirical data necessary for reproducibility. Of the 172 publications with empirical data, 43 (25%) reported data availability statements, 4 (2.33%) analysis scripts, 4 (2.33%) links to a protocol, and 10 (5.81%) were pre-registered. Conclusion:Our study found that reproducible and transparent research practices are infrequently employed by the nephrology research community. Greater efforts should be made by both funders and journals, two entities that have the greatest ability to influence change. In doing so, an open science culture may eventually become the norm rather than the exception. Introduction:Reproducibility is critical to diagnostic accuracy and treatment implementation. In nephrology, a substantial body of literature is devoted to establishing the reproducibility of diagnostic tests or procedures. Examples include an evaluation of the reproducibility of the Banff classification for surveillance renal allograft biopsies among pathologists across transplant centers 1 , a novel analytic technique for renal blood oxygenation level-dependent MRI 2 , and a food frequency questionnaire among patients with chronic kidney disease 3 . This form of reproducibility is important clinically, as such studies establish our confidence in tests or procedures for applications to patient care.Concurrent with clinical reproducibility, research reproducibility establishes whether use of identical study materials and methodologies in replication efforts permit researchers to arrive at similar results and conclusions. In other cases, reproducibility may mean attempts to reanalyze study data to determine whether the same results can be obtained. The National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) supports the National Institutes of Health's (NIH) rigor and reproducibility initiative, which was created to foster greater reproducibility of studies funded by taxpayer dollars. The NIDDK also sponsors dkNET 4 , a porta...

How often do cancer researchers make their data and code available and what factors are associated with sharing?

“…Specifically, we noted that only one of the 59 articles that declared data was available complied with our FAIR assessment. This observation, depending on how availability for reuse is defined, is unfortunately consistent with this body of research which has reported 50-100% reductions in availability following interrogation of sharing statements 15-25 ; with factors such as the lack of unique and permanent identifiers, meta-data and licensing terms being noted as major pitfalls. 30,47 Furthermore, while we also noted a strong relationship between mandatory data sharing policies and actual data availability, we unfortunately also observed similarly sub-optimal compliance with these policies too; a finding that has been noted by other studies both inside and outside of medicine.…”

“…Rather our observations that 19% and 4% of cancer researchers declared data and code were publicly available respectively are consistent with several studies reporting low, but increasing, declaration rates ranging between 3-24% and 0-2% respectively over several medical disciplines between 2014-2018. [15][16][17][18][19][20][21][22][23][24][25] The increase of declarations over time -particularly data availability declarations -is likely due to the growing number of medical journals that are adopting stronger policies on data and code sharing, particularly those that are requiring the addition of availability statements. For example, we note that a quarter of the unique 235 journals analysed in our study had adopted a mandatory data sharing policy for some or all data, which is higher than a previous survey of medical journal editors in the previous year.…”

“…14 While significant progress has been made towards increasing the availability of the products of research (i.e., data, code and materials), previous research on the topic has reported low availability of data (0-6%) and code (0-2%) across many medical fields between 2014 and 2018. [15][16][17][18][19][20][21][22][23][24][25] Furthermore, other research in medicine has shown sub-optimal compliance with journal data and code sharing policies. [26][27][28] In this study we build on previous research to investigate how frequently cancer researchers share the data underlying their research, as well as the code used to perform statistical analyses in a large random sample of published cancer studies.…”

Authorial Conflicts of Interest and Sponsorship in Systematic Reviews and Meta-Analyses on Psoriasis

“…The reproducibility of systematic reviews, however, is very difficult if not impossible to undertake. The senior author of this study widely evaluates research reproducibility [41][42][43][44][45][46][47][48][49] and has evaluated the reproducibility of meta-analyses. 50 While meta-analytic effect estimates may be more reproducible in some cases, earlier steps of the systematic review process are likely not.…”

Toward Transparency in Nephrology Research

“…S3 Fladie et al have evaluated the nephrology literature for indicators of transparency and reliability of research. 9 The authors have gone to lengths to make the process for this article transparent and reproducible. They randomly sampled 172 articles from English language nephrology over 5 years (2014À2018).…”