Replications, Comparisons, Sampling and the Problem of Representativeness in Animal Cognition Research

Abstract

Animal cognition research often involves small and idiosyncratic samples. This can constrain the generalizability and replicability of a study’s results and prevent meaningful comparisons between samples. However, there is little consensus about what makes a strong replication or comparison in animal research. We apply a resampling definition of replication to answer these questions in Part 1 of this article, and, in Part 2, we focus on the problem of representativeness in animal research. Through a case study and a simulation study, we highlight how and when representativeness may be an issue in animal behavior and cognition research and show how the representativeness problems can be viewed through the lenses of, i) replicability, ii) generalizability and external validity, iii) pseudoreplication and, iv) theory testing. Next, we discuss when and how researchers can improve their ability to learn from small sample research through, i) increasing heterogeneity in experimental design, ii) increasing homogeneity in experimental design, and, iii) statistically modeling variation. Finally, we describe how the strongest solutions will vary depending on the goals and resources of individual research programs and discuss some barriers towards implementing them.

The Status and Value of Replications in Animal Behavior Science

“…Thus, at a minimum, all experiments are a test of an effect's generalizability over time. The resampling approach thus illustrates how the question of replicability is always a question of generalizability (Farrar et al, 2021b). The link between replication and generalizability is further highlighted when the terminology from the resampling account is linked with statistical terminology (as explained in Halina, 2021)a token that is resampled is one that is treated as a random factor (Machery, 2020) and thus stands as an example of a broader class (see also Yarkoni, 2020).…”

“…Animal Behavior and Cognition specifically states that empirical replication studies will be published, and authors are encouraged to use the Pre-registered Replication Articles format the journal offers (Beran, 2020). Researchers in the field seem to recognize the need for replication studies (Farrar et al, 2021b;Fraser et al, 2020), and multiple publications discuss recommendations for increasing replicability of specific fields (Farrar et al, 2020, for comparative cognition;O'Dea et al, 2021, for ecology and evolutionary biology).…”

“…Researchers may be able to employ multiple approaches in a field with theoretical openness. In some research areas, the theoretical openness may be decreased through theory development, aided by formalization (Allen, 2014;Farrar et al, 2021b;Guest & Martin, 2020;Lee et al, 2019;Lind, 2018;Smith et al, 2012;van Rooij & Baggio, 2021; for a study that set out to reproduce a model of ant nest wall building in this issue see Invernizzi & Ruxton, 2021). In areas in which this is not possible, claims may have to be adjusted to reflect the epistemic circumstances (Yarkoni, 2020) and criteria other than replications can be employed to evaluate the quality of research and the reliability of claims (Leonelli, 2018).…”

“…Similarly, while 85% of researchers agreed that conducting replications in animal behavior and cognition research is important, the majority of surveyed researchers (70%) also reported that too few studies are explicitly replicating previous work (Farrar et al, 2021a). A range of different factors explaining this (apparent) lack of replications are discussed in the theoretical contributions in this special issue, such as reluctance of funders to finance them, time and resource constraints, and incentive structures (Farrar et al, 2021b;Khan & Wascher, 2021;Shaw et al, 2021).…”

Doing reliable research in comparative psychology: Challenges and proposals for improvement.

“…It is important to bear in mind that replication should be viewed as a process of evidence-building, rather than individual replications being considered as successes or failures (Edlund et al, 2021). There are many reasons that a comparative psychology replication study might not produce the same findings, aside from the possibility that the initial findings were spurious, such as seasonal and developmental variation in behaviour (Farrar et al, 2021). In addition, the earlier section on Comparing across testing sites highlighted numerous variables that can influence animals' performance in experimental tasks, including experimenter, housing arrangements, and prior experience.…”