The study design elements employed by researchers in preclinical animal experiments from two research domains and implications for automation of systematic reviews

Abstract

Systematic reviews are increasingly using data from preclinical animal experiments in evidence networks. Further, there are ever-increasing efforts to automate aspects of the systematic review process. When assessing systematic bias and unit-of-analysis errors in preclinical experiments, it is critical to understand the study design elements employed by investigators. Such information can also inform prioritization of automation efforts that allow the identification of the most common issues. The aim of this study was to identify the design elements used by investigators in preclinical research in order to inform unique aspects of assessment of bias and error in preclinical research. Using 100 preclinical experiments each related to brain trauma and toxicology, we assessed design elements described by the investigators. We evaluated Methods and Materials sections of reports for descriptions of the following design elements: 1) use of comparison group, 2) unit of allocation of the interventions to study units, 3) arrangement of factors, 4) method of factor allocation to study units, 5) concealment of the factors during allocation and outcome assessment, 6) independence of study units, and 7) nature of factors. Many investigators reported using design elements that suggested the potential for unit-of-analysis errors, i.e., descriptions of repeated measurements of the outcome (94/200) and descriptions of potential for pseudo-replication (99/200). Use of complex factor arrangements was common, with 112 experiments using some form of factorial design (complete, incomplete or split-plot-like). In the toxicology dataset, 20 of the 100 experiments appeared to use a split-plot-like design, although no investigators used this term. The common use of repeated measures and factorial designs means understanding bias and error in preclinical experimental design might require greater expertise than simple parallel designs. Similarly, use of complex factor arrangements creates novel challenges for accurate automation of data extraction and bias and error assessment in preclinical experiments.

Reporting and analysis of repeated measurements in preclinical animals experiments

“…Fifth-eight of 200 articles met the criteria of repeated measures design element [4], i.e., used language that suggested repeated measurement of the outcome on study subjects. As summarized in Table 2, only three of the 58 studies included a rationale of a repeated measurement of the outcome.…”

“…Repeated measures on the outcome is a commonly used design element in preclinical animal experiments [4]. The rationale for incorporating a repeated measurement design element into an experiment is generally considered to be of interest in a treatment-by-time interaction or effective control of subject-to-subject variation [5].…”

“…The 200 studies are preclinical experiments on brain trauma and toxicology, many with repeated measure design elements that were identified for a different study. Eligible studies included described primary research of a single comparative pre-clinical animal experiment [4]. Of the 200 studies, 100 studies were randomly selected from eligible studies and characterized for the design features used by the authors.…”

“…Repeated measures is a very common study design element in preclinical animal research. In a survey of 200 preclinical animal experiments, 58 manuscripts contained language that described the repeated measurement of at least one outcome on study subjects [4]. Inclusion of a repeated measurements design element can be desirable for a variety of reasons.…”

Guidelines for repeated measures statistical analysis approaches with basic science research considerations

“…Study designs with repeated measurements are common in basic science research. A systematic review of preclinical animal research studies found that approximately 50% of studies within biomedical research domains of toxicology and brain trauma reported a study design with repeated measurements ( 3 ). Planned statistical analyses for this type of research design should incorporate all observed repeated measurements to evaluate the research hypotheses.…”