Don’t Forget the Psychology in Analyses of Psychological Data: The Case of Sequential Testing

Abstract

Sequential testing enables researchers to monitor and analyze data as it arrives, and decide whether or not to continue data collection depending on the results. Bayesian approaches can mitigate many statistical issues with sequential testing, but we suggest that current discussions of the topic are limited by focusing almost entirely on the mathematical underpinnings of analytic approaches. An important but largely neglected assumption of sequential testing is that the data generating process under investigation remains constant across the experimental cycle. Without care, psychological factors may result in violations of this assumption when sequential testing is used: researchers’ behavior may be changed by the observation of incoming data, in turn influencing the process under investigation. We urge that researchers adopt an ‘insulated’ sequential testing approach, in which research personnel remain blind to the results of interim analyses. We discuss different ways of achieving this, from automation to collaborative inter-lab approaches. As a practical supplement to the issues we raise, we introduce an evolving resource for sequential testing – the Sequential Testing Hub (www.sequentialtesting.com) – including templates and a guide for involving an independent analyst in a sequential testing pipeline, an annotated bibliography centered around the practicalities of sequential analyses, and space for suggestions to help develop this resource further. We aim to show that although unfettered use of sequential testing may raise problems, carefully designed procedures can limit the pitfalls arising from its use, allowing researchers to capitalize on the benefits it provides.

Powerful sequential designs using Bayesian estimation: A power analysis tutorial using brms, the tidyverse, and furrr

“…Researchers would be wise to keep in mind that taking care of statistical aspects of study design does not secure the study's success or integrity. A range of psychological factors may jeopardize even properly-powered studies (Elsey et al, 2021). In sequential analysis, one crucial assumption is that the data-generating process remains constant across the experimental cycle.…”

“…Researchers are advised to carefully consider how knowledge of interim effects might impact upon the datagenerating process, including researchers consciously or unconsciously pushing participants in a particular direction, but also simply affecting the motivation and diligence with which the researcher engages with participants. Various means of insulating sequential analyses from the possible psychological impact of interim data checks might be pursued (Elsey et al, 2021). Possibilities include automating analyses such that only information about whether to cease or continue data collection is presented to researchers (Beffara-Bret et al, 2019), or involving analysts who remain independent of the researchers who interact with participants/patients, in a similar vein to data monitoring committees often used in medical trials (Friedman et al, 2015;Lakens, 2014).…”