Reproducibility and replicability of rodent phenotyping in preclinical studies

Kafkafi, Neri; Agassi, Joseph; Chesler, Elissa J.; Crabbe, John C.; Crusio, Wim E.; Eilam, David; Golani, Ilan; Gomez‐Marin, Alex; Heller, Ruth; Iraqi, Fuad A.; Jaljuli, Iman; Karp, Natasha A.; Morgan, Hugh W.; Nicholson, George; Pfaff, Donald W.; Richter, S. Helene; Stark, Philip B.; Stiedl, Oliver; Stodden, Victoria; Tarantino, Lisa M.; Tucci, Valter; Valdar, William; Williams, Robert W.; Würbel, Hanno; Benjamini, Yoav

doi:10.1101/079350

Cited by 16 publications

(19 citation statements)

References 83 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This raises important questions: To what extent might these represent false positive results? Could widespread use of EWOCs account for why there are such challenges in finding reproducible phenotypes in behavioral models (Kafkafi et al, 2018)? In order to determine how vulnerable this approach is to false positive interpretations, we conducted extensive simulation studies as detailed below.…”

Section: Resultsmentioning

confidence: 99%

Erroneous inference based on a lack of preference within one group: autism, mice, and the Social Approach Task

Nygaard¹,

Maloney²,

Dougherty³

2019

Preprint

View full text Add to dashboard Cite

The Social Approach Task is commonly used to identify sociability deficits when modeling liability factors for autism spectrum disorder (ASD) in mice. It was developed to expand upon assays available to examine distinct aspects of social behavior in rodents and has become a standard component of mouse ASD-relevant phenotyping pipelines. However, there is variability in the statistical analysis and interpretation of results from this task. A common analytical approach is to conduct within-group comparisons only, and then interpret a difference in significance levels as if it were a group difference, without any direct comparison. As an efficient shorthand, we named this approach EWOCs:Erroneous Within-group Only Comparisons. Here we examined the prevalence of EWOCs and used simulations to test whether it could produce misleading inferences. Our review of Social Approach studies of high-confidence ASD genes revealed 45% of papers sampled used only this analytical approach. Through simulations, we then demonstrate how a lack of significant difference within one group often doesn’t correspond to a significant difference between groups, and show this erroneous interpretation increases the rate of false positives up to 25%. Finally, we define a simple solution: use an index, like a social preference score, with direct statistical comparisons between groups to identify significant differences. We also provide power calculations to guide sample size in future studies. Overall, elimination of EWOCs and adoption of direct comparisons should result in more accurate, reliable, and reproducible data interpretations from the Social Approach Task across ASD liability models.Lay SummaryThe Social Approach Task is widely used to assess social behavior in mice and is frequently used in studies modeling autism. However, reviewing published studies showed nearly half do not use correct comparisons to interpret the data. Using simulated and original data, we argue the correct statistical approach is a direct comparison of scores between groups. This simple solution should reduce false positives and improve consistency of results across studies.

show abstract

Section: Resultsmentioning

confidence: 99%

Erroneous inference based on a lack of preference within one group: autism, mice, and the Social Approach Task

Nygaard¹,

Maloney²,

Dougherty³

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Second, replicates provide not only an insurance policy against phenotyping errors, but also a way to average over batches and similar nuisance parameters (Cowen 1988), thus protecting against the negative consequences of gene by environment interactions while also providing the opportunity for such interactions to be detected [e.g., Kafkafi et al (2005Kafkafi et al ( , 2018].…”

Section: More Strains Vs More Replicatesmentioning

confidence: 99%

Determinants of QTL mapping power in the realized Collaborative Cross

Keele

Crouse

Kelada

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

The Collaborative Cross (CC) is a mouse genetic reference population whose range of applications includes quantitative trait loci (QTL) mapping. The design of a CC QTL mapping study involves multiple decisions, including which and how many strains to use, and how many replicates per strain to phenotype, all viewed within the context of hypothesized QTL architecture. Until now, these decisions have been informed largely by early power analyses that were based on simulated, hypothetical CC genomes. Now that more than 50 CC strains are available and more than 70 CC genomes have been observed, it is possible to characterize power based on realized CC genomes. We report power analyses based on extensive simulations and examine several key considerations: 1) the number of strains and biological replicates, 2) the QTL effect size, 3) the presence of population structure, and 4) the distribution of functionally distinct alleles among the founder strains at the QTL.We also provide general power estimates to aide in the design of future experiments. All analyses were conducted with our R package, SPARCC (Simulated Power Analysis in the Realized Collaborative Cross), developed for performing either large scale power analyses or those tailored to particular CC experiments. KEYWORDS recombinant inbred lines, haplotype association, allelic series, multiparental population, MPP, quantitative trait, complex trait 32 et al. 2014) 33 Nonetheless, QTL mapping power depends in part on the 34 number of strains available, and the number strains available 35 in the CC is, and will remain, far less than the 1,000 proposed 36 in Churchill et al. (2004): At the time of this work, mice were 37 QTL mapping power in Collaborative Cross 1 available for 59 CC strains from the UNC Systems Genetics Core, 38 with a subset from these 59 and an additional 11 expected to be 39 offered through the Jackson Laboratory (JAX), a total of 70 CC 40 strains potentially. 41 A reduction in strain numbers as a function of allelic incom-42 patibilities between subspecies (Shorter et al. 2017) was expected, 43 and winnowed the number of resulting CC strains down to 50-44 70. Although smaller than originally intended, this population 45 size reflects the biological and financial realities of maintaining a 46 sustainable mammalian genome reference population. [Whereas 47 129 3. Evaluation of QTL detection accuracy, power and false pos-130itive rate (FPR). 131These are described in detail below, after a description of the 132 genomic data that serves as the basis for the simulations.

show abstract

“…Recent discussion of replication and reliability across scientific disciplines (Coiera, Ammenwerth, Georgiou, & Magrabi, 2018;Davis-Kean & Ellis, 2019;Gerlai, 2019;Kafkafi et al, 2018;Waters, LeBeau, Young, Dowell, & Ryan, 2019) identifies the generalization of effects, beyond narrowly constrained protocols, as a key issue. It may be the case that what appear to be minor variations from a published method are in fact important and the target effect(s) do(es) not generalize across such variations.…”

Section: Introductionmentioning

confidence: 99%

Effects of Δ⁹-tetrahydrocannabinol (THC) vapor inhalation in Sprague-Dawley and Wistar rats.

Taffe¹,

Creehan²,

Vandewater³

et al. 2021

Experimental and Clinical Psychopharmacology

View full text Add to dashboard Cite

An inhalation system based on e-cigarette technology produces hypothermic and anti-nociceptive effects of ∆ 9-tetrahydrocannabinol (THC) in rats. Indirect comparison of some prior investigations suggested differential impact of inhaled THC between Wistar (WI) and Sprague-Dawley (SD) rats, thus this study was conducted to directly compare the strains across inhaled and injected routes of administration. Groups (N=8 per strain) of age matched male SD and WI rats were prepared with radiotelemetry devices to measure temperature and then exposed to vapor from the propylene glycol (PG) vehicle or THC (25-200 mg/mL of PG) for 30 or 40 minutes. Additional studies evaluated effects of THC inhalation on plasma THC (50-200 mg/mL) and nociception (100-200 mg/mL) as well as the thermoregulatory effect of intraperitoneal injection of THC (5-30 mg/kg). Hypothermic effects of THC were more pronounced in SD rats, where plasma levels of THC were identical across strains, under either fixed inhalation conditions or injection of a mg/kg equivalent dose. Strain differences in hypothermia were largest after i.p. injection of THC with SD rats exhibiting dosedependent temperature reduction after 5 or 10 mg/kg, i.p. and the WI rats only exhibiting significant hypothermia after 20 mg/kg, i.p. The anti-nociceptive effects of inhaled THC (100, 200 mg/mL) did not differ significantly across the strains. These studies confirm an insensitivity of WI rats, compared with SD rats, to hypothermia induced by THC following inhalation conditions that produced identical plasma THC and anti-nociception. Thus, quantitative, albeit not qualitative, strain differences may obtain when studying thermoregulatory effects of THC.

show abstract

Reproducibility and replicability of rodent phenotyping in preclinical studies

Cited by 16 publications

References 83 publications

Erroneous inference based on a lack of preference within one group: autism, mice, and the Social Approach Task

Erroneous inference based on a lack of preference within one group: autism, mice, and the Social Approach Task

Determinants of QTL mapping power in the realized Collaborative Cross

Effects of Δ⁹-tetrahydrocannabinol (THC) vapor inhalation in Sprague-Dawley and Wistar rats.

Contact Info

Product

Resources

About