Do multiple experimenters improve the reproducibility of animal studies?

Abstract

The credibility of scientific research has been seriously questioned by the widely claimed “reproducibility crisis”. In light of this crisis, there is a growing awareness that the rigorous standardisation of experimental conditions may contribute to poor reproducibility of animal studies. Instead, systematic heterogenisation has been proposed as a tool to enhance reproducibility, but a real-life test across multiple independent laboratories is still pending. The aim of this study was therefore to test whether heterogenisation of experimental conditions by using multiple experimenters improves the reproducibility of research findings compared to standardised conditions with only one experimenter. To this end, we replicated the same animal experiment in 3 independent laboratories, each employing both a heterogenised and a standardised design. Whereas in the standardised design, all animals were tested by a single experimenter; in the heterogenised design, 3 different experimenters were involved in testing the animals. In contrast to our expectation, the inclusion of multiple experimenters in the heterogenised design did not improve the reproducibility of the results across the 3 laboratories. Interestingly, however, a variance component analysis indicated that the variation introduced by the different experimenters was not as high as the variation introduced by the laboratories, probably explaining why this heterogenisation strategy did not bring the anticipated success. Even more interestingly, for the majority of outcome measures, the remaining residual variation was identified as an important source of variance accounting for 41% (CI95 [34%, 49%]) to 72% (CI95 [58%, 88%]) of the observed total variance. Despite some uncertainty surrounding the estimated numbers, these findings argue for systematically including biological variation rather than eliminating it in animal studies and call for future research on effective improvement strategies.

Mapping strategies towards improved external validity in preclinical translational research

“…Experimental conditions experimenter 0 14 (20,25,26,(31)(32)(33)(34)(35)45,46,48,49,54,58,59) Yes (25,32,34,45,46,54,58,59) time of day 4 (9,19,28,29) 12 (14,21,26,(31)(32)(33)(34)(35)46,48,49,57) Yes (21,32,34,46) test conditions 6 (9,10,26,27,45,47) 6 (11,14,30,35,…”

“…Regarding experimental conditions, the impact of experimenters is a relevant source of variability particularly for behavioral outcomes and has been directly considered in studies of systematic heterogenization. The evidence, however, is ambiguous (25,46,54,58,59). One possibility is that each tested experimenter represents a conflation of uncountable features that could theoretically have an impact.…”

“…Most research with a translational focus is not interested in the impact of having specific experimenters nor the variability between them. Rather, having multiple experimenters involved has been proposed as a surrogate for when multicenter studies are not possible as it is hypothesized to be a major source of variation in such studies (9,58).…”

Planning preclinical confirmatory multicenter trials to strengthen translation from basic to clinical research – a multi-stakeholder workshop report

“…One naturally occurring source of variation is the different experimenters themselves. However, the latest literature indicates that this is less of an issue, particularly if all involved parties are well trained [38,39]. To assess replicability of results across centers, a low number of centers already is sufficient.…”

Behavioural Voluntary and Social Bioassays Enabling Identification of Complex and Sex-Dependent Pain- (-Related) Phenotypes in Rats with Bone Cancer

“…In this context, monitoring behavior in home cage settings seems promising to broaden behavioral studies and investigate the animals within their familiar environments. In this way, the physical contact between the animals and the human experimenters is reduced, thereby minimizing any direct experimenter effects ( Kortzfleisch et al, 2022; Sadler et al, 2021; Sorge et al, 2014 ). At the same time, it might become possible to detect more subtle changes in behavior that indicate pain-related behavior in more detail and/or with more reliability ( Deuis et al, 2017; González-Cano et al, 2020; Mogil, 2015; Tappe-Theodor et al, 2019 ).…”