Outcome heterogeneity and bias in acute experimental spinal cord injury

Abstract

ObjectiveTo determine whether and to what degree bias and underestimated variability undermine the predictive value of preclinical research for clinical translation. MethodsWe investigated experimental spinal cord injury (SCI) studies for outcome heterogeneity and the impact of bias. Data from 549 preclinical SCI studies including 9,535 animals were analyzed with meta-regression to assess the effect of various study characteristics and the quality of neurologic recovery. ResultsOverall, the included interventions reported a neurobehavioral outcome improvement of 26.3% (95% confidence interval 24.3-28.4). Response to treatment was dependent on experimental modeling paradigms (neurobehavioral score, site of injury, and animal species). Applying multiple outcome measures was consistently associated with smaller effect sizes compared with studies applying only 1 outcome measure. More than half of the studies (51.2%) did not report blinded assessment, constituting a likely source of evaluation bias, with an overstated effect size of 7.2%. Assessment of publication bias, which extrapolates to identify likely missing data, suggested that between 2% and 41% of experiments remain unpublished. Inclusion of these theoretical missing studies suggested an overestimation of efficacy, reducing the effect sizes by between 0.9% and 14.3%. ConclusionsWe provide empirical evidence of prevalent bias in the design and reporting of experimental SCI studies, resulting in overestimation of the effectiveness. Bias compromises the internal validity and jeopardizes the successful translation of SCI therapies from the bench to bedside. Glossary BBB = Basso, Beattie, Bresnahan; BMS = Basso Mouse Scale for Locomotion; CAMARADES = Collaborative Approach to Meta-Analysis and Review of Animal Data From Experimental Studies; CI = confidence interval; OEG = olfactory ensheathing glia; SCI = spinal cord injury.To allow a more homogeneous group characterized by similar effect sizes (ESs), a rodent analysis was completed identifying the Basso, Beattie, and Bresnahan (BBB)/Basso Mouse Scale (BMS) only group as comparable with regard to the main outcome parameter (improved locomotor function, figure 1B, red square). Funnel plots illustrating the precision (1 divided by the SEM, y-axis) plotted against the standardized ES (x-axis). Each black dot represents 1 experiment. In the absence of publication bias, the points should resemble an inverted funnel. Black dotted line indicates the reported ES before inclusion of missing studies; red line, after trim and till analysis. (A) Decompression: reported ES 27.1% (95% confidence interval [CI] 15.1-39.1, n = 22 studies), adjusted ESs 12.8% (95% CI 1.7-24.0, n = 31). (B) Exercise: reported ES 14.4% (95% CI 6.4-22.3, n = 35), adjusted ES 1.9% (95% CI −6.9 to 10.7, n = 42). (C) Hypothermia: reported ES 16.3% (95% CI 10.3-22.3) without adjustments. (D) Olfactory ensheathing glia (OEG): reported ES 20.1% (95% CI 15.2-24.9, n = 53), adjusted ES 19.2 (95% CI 14.4-24.0, n = 54). (E) Rho/ROCK inhibitors: reported ES 21...

Weight loss, insulin resistance, and study design confound results in a meta-analysis of animal models of fatty liver

“…Even though we found evidence for efficacy of the majority of drugs included in this analysis, the 95% CI for treatment effect size overlapped for most drug classes. This is generally consistent with findings reported in preclinical models of spinal cord injury where the effect size of several different types of treatment overlapped ( Watzlawick et al, 2019 ). Overall, the trends observed are not consistent with findings in humans and there does not appear to be any clear patterns that indicate potentially successful translation.…”

“…Systematic analyses of preclinical studies have found that publication bias may account for at least a third of the estimate of efficacy in trials ( Henderson et al, 2015 ; Sena et al, 2010 ; van der Worp et al, 2010 ). In addition, other variables of animal model design can influence the magnitude of the treatment response ( Watzlawick et al, 2019 ) and reporting of model design is often incomplete ( Flórez-Vargas et al, 2016 ). These findings are highly relevant in the context of the ‘reproducibility crisis’ ( Baker, 2016 ; von Herrath et al, 2019 ) as well as having ethical implications for the use of animals in research that is not of optimum quality ( Prescott and Lidster, 2017 ).…”

What influences treatment response in animal models of non-alcoholic fatty liver disease? A meta-analysis with meta-regression

“…To our knowledge this is such similarity between multiple drugs in preclinical models has not been described before to this extent. It is generally consistent with findings reported in preclinical models of spinal cord injury where the effect size of several different types of treatment overlapped [11] .…”

“…Systematic analyses of preclinical studies, predominantly in the field of neuroscience, have found that publication bias may account for at least a third of the estimate of efficacy in trials [9,10] . In addition, other variables of animal model design can influence the magnitude of the treatment response [11] . These findings are highly relevant in the context of the 'reproducibility crisis' [12,13] as well as having ethical implications of the use of animals in research that is not of optimum quality [14] .…”

Measuring context-dependent memory in rodents: a systematic review and meta-analysis of important variables in the object-in-context task

“…Still, lack of reporting introduces an unclear risk and at least difficulty to estimate the quality of the studies [55]. The problem of insufficient reporting of experimental details and quality measures in pre-clinical studies has been extensively addressed and previously observed in metaanalysis [35,[56][57][58][59][60].…”