Establishing ground truth in the clinical neurosciences: if replication is the answer, then what are the questions?

Abstract

The replication crisis poses important challenges to modern science. Central to this challenge is re-establishing ground truths, or the most fundamental theories that serve as the bedrock to a scientific community. However, the goal to identify hypotheses with the greatest support is nontrivial given the current unprecedented rate of scientific publishing. In this era of high-volume science, the goal of this study is to sample from one research community within clinical neuroscience (traumatic brain injury; TBI) and track major trends that have shaped this literature over the past 50 years. To do so, we first conduct decade-wise (1980-2019) network analysis to examine the scientific communities that shape this literature. To establish the robustness of our network findings, we utilized searches from separate search engines (Web of Science; Semantic Scholar). As a second goal, we sought to determine the most highly cited hypotheses influencing the literature in each decade. In a third goal, we then searched for any papers referring to “replication” or efforts to reproduce findings within our >50,000 paper dataset. From this search, 550 papers were analyzed to determine the frequency and nature of formal replication studies over time. Finally, to maximize transparency, we provide a detailed procedure for the creation and analysis of our dataset, including a discussion of each of our major decision points, to facilitate similar efforts in other areas of neuroscience. We found that the unparalleled rate of scientific publishing within the TBI literature combined with the scarcity of clear hypotheses in individual publications are a challenge to both evaluating accepted findings and determining future directions. Additionally, while the conversation about reproducibility has increased over the past decade, the rate of published replication studies continues to be a negligible proportion of the research. Meta-science and computational methods offer the critical opportunity to assess the state of the science and illuminate pathways forward, but ultimately there is structural change needed in the TBI literature and perhapsothers.

How failure to falsify in high-volume science contributes to the replication crisis

“…The number of papers published per year continues to increase without asymptote ( Bornmann and Mutz, 2015 ). When considering all papers published in the clinical neurosciences since 1900, more than 50% of the entire literature has been published in the last 10 years and 35% in the last five years (see supplementary figures S1a,b in Priestley et al, 2022 ). In the most extreme examples, “hyperprolific” lab directors publish a scientific paper roughly every 5 days ( Ioannidis et al, 2018 ).…”