Systematic assessment of the replicability and generalizability of preclinical findings: Impact of protocol harmonization across laboratory sites

Abstract

The influence of protocol standardization between laboratories on their replicability of preclinical results has not been addressed in a systematic way. While standardization is considered good research practice as a means to control for undesired external noise (i.e., highly variable results), some reports suggest that standardized protocols may lead to idiosyncratic results, thus undermining replicability. Through the EQIPD consortium, a multi-lab collaboration between academic and industry partners, we aimed to elucidate parameters that impact the replicability of preclinical animal studies. To this end, 3 experimental protocols were implemented across 7 laboratories. The replicability of results was determined using the distance travelled in an open field after administration of pharmacological compounds known to modulate locomotor activity (MK-801, diazepam, and clozapine) in C57BL/6 mice as a worked example. The goal was to determine whether harmonization of study protocols across laboratories improves the replicability of the results and whether replicability can be further improved by systematic variation (heterogenization) of 2 environmental factors (time of testing and light intensity during testing) within laboratories. Protocols were tested in 3 consecutive stages and differed in the extent of harmonization across laboratories and standardization within laboratories: stage 1, minimally aligned across sites (local protocol); stage 2, fully aligned across sites (harmonized protocol) with and without systematic variation (standardized and heterogenized cohort); and stage 3, fully aligned across sites (standardized protocol) with a different compound. All protocols resulted in consistent treatment effects across laboratories, which were also replicated within laboratories across the different stages. Harmonization of protocols across laboratories reduced between-lab variability substantially compared to each lab using their local protocol. In contrast, the environmental factors chosen to introduce systematic variation within laboratories did not affect the behavioral outcome. Therefore, heterogenization did not reduce between-lab variability further compared to the harmonization of the standardized protocol. Altogether, these findings demonstrate that subtle variations between lab-specific study protocols may introduce variation across independent replicate studies even after protocol harmonization and that systematic heterogenization of environmental factors may not be sufficient to account for such between-lab variation. Differences in replicability of results within and between laboratories highlight the ubiquity of study-specific variation due to between-lab variability, the importance of transparent and fine-grained reporting of methodologies and research protocols, and the importance of independent study replication.

A systematic assessment of preclinical multilaboratory studies and a comparison to single laboratory studies

“…Alternatively, labs developed a general protocol but adapted it to fit their own respective budget with what resources they had. Of note, recent work has suggested that harmonization reduces between-lab variability, however, systematic heterogenization did not reduce variability further ( Arroyo-Araujo et al, 2022 ); this may suggest that, even in fully harmonized protocols, enough uncontrolled heterogeneity exists that further purposeful heterogenization has little effect. Another barrier identified was ethics approval for animal experiments at all the labs ( Llovera et al, 2015 ).…”

A Systematic Assessment of Robustness in CNS Safety Pharmacology

“…The multinational EQIPD consortium ( E nhanced Q uality in P reclinical D ata, https://quality-preclinical-data.eu/) aimed to identify factors influencing the quality of data generated in preclinical research as a basis for recommendations enabling a smoother and successful transition from preclinical research to clinical application (9, 10). In a recent study, the consortium analyzed the impact of protocol harmonization on the replicability of data in the open field test, i.e., a behavioral paradigm with automated recording of the primary outcome parameters (11). The study demonstrated that harmonization of protocols can reduce between-site variability (11).…”

“…In a recent study, the consortium analyzed the impact of protocol harmonization on the replicability of data in the open field test, i.e., a behavioral paradigm with automated recording of the primary outcome parameters (11). The study demonstrated that harmonization of protocols can reduce between-site variability (11). Considering that the standardized application of scoring systems and their harmonization across sites can pose a particular challenge, we next addressed the question of whether a comparable impact can be observed for a frequently used paradigm, based on scoring of a variety of readout parameters by experimenters.…”

“…To allow comparability with two other subprojects of the EQIPD consortium that focused on an open field paradigm (11) and pharmaco-EEG, the study was performed in mice, representing the second most common species for Irwin/FOB testing (16, 24). MK-801 was chosen as the compound, considering its broad range of effects on various parameters from different functional domains assessed in the Irwin test.…”