Sequential testing enables researchers to monitor and analyze data as it arrives, and decide whether or not to continue data collection depending on the results. Although there are approaches that can mitigate many statistical issues with sequential testing, we suggest that current discussions of the topic are limited by focusing almost entirely on the mathematical underpinnings of analytic approaches. An important but largely neglected assumption of sequential testing is that the data generating process under investigation remains constant across the experimental cycle. Without care, psychological factors may result in violations of this assumption when sequential testing is used: researchers’ behavior may be changed by the observation of incoming data, in turn influencing the process under investigation. We argue for the consideration of an ‘insulated’ sequential testing approach, in which research personnel remain blind to the results of interim analyses. We discuss different ways of achieving this, from automation to collaborative inter-lab approaches. As a practical supplement to the issues we raise, we introduce an evolving resource aimed at helping researchers navigate both the statistical and psychological pitfalls of sequential testing: the Sequential Testing Hub (www.sequentialtesting.com). The site includes a guide for involving an independent analyst in a sequential testing pipeline, an annotated bibliography of relevant articles covering statistical aspects of sequential testing, links to tools and tutorials centered around how to actually implement a sequential analysis in practice, and space for suggestions to help develop this resource further. We aim to show that although unfettered use of sequential testing may raise problems, carefully designed procedures can limit the pitfalls arising from its use, allowing researchers to capitalize on the benefits it provides.
Background Alcohol use disorders can be conceptualised as a learned pattern of maladaptive alcohol-consumption behaviours. The memories encoding these behaviours centrally contribute to long-term excessive alcohol consumption and are therefore an important therapeutic target. The transient period of memory instability sparked during memory reconsolidation offers a therapeutic window to directly rewrite these memories using targeted behavioural interventions. However, clinically-relevant demonstrations of the efficacy of this approach are few. We examined key retrieval parameters for destabilising naturalistic drinking memories and the ability of subsequent counterconditioning to effect long-term reductions in drinking. Methods Hazardous/harmful beer-drinking volunteers (N = 120) were factorially randomised to retrieve (RET) or not retrieve (No RET) alcohol reward memories with (PE) or without (No PE) alcohol reward prediction error. All participants subsequently underwent disgust-based counterconditioning of drinking cues. Acute responses to alcohol were assessed pre- and post-manipulation and drinking levels were assessed up to 9 months. Results Greater long-term reductions in drinking were found when counterconditioning was conducted following retrieval (with and without PE), despite a lack of short-term group differences in motivational responding to acute alcohol. Large variability in acute levels of learning during counterconditioning was noted. ‘Responsiveness’ to counterconditioning predicted subsequent responses to acute alcohol in RET + PE only, consistent with reconsolidation-update mechanisms. Conclusions The longevity of behavioural interventions designed to reduce problematic drinking levels may be enhanced by leveraging reconsolidation-update mechanisms to rewrite maladaptive memory. However, inter-individual variability in levels of corrective learning is likely to determine the efficacy of reconsolidation-updating interventions and should be considered when designing and assessing interventions.
Disrupting memory reconsolidation provides an opportunity to abruptly reduce the behavioural expression of fear memories with long-lasting effects. The success of a reconsolidation intervention is, however, not guaranteed as it strongly depends on the destabilization of the memory. Identifying the necessary conditions to trigger destabilization remains one of the critical challenges in the field. We aimed to replicate a study from our lab, showing that the occurrence of a prediction error (PE) during reactivation is necessary but not sufficient for destabilization. We tested the effectiveness of a reactivation procedure consisting of a single PE, compared to two control groups receiving no or multiple PEs. All participants received propranolol immediately after reactivation and were tested for fear retention 24 h later. In contrast to the original results, we found no evidence for a reconsolidation effect in the single PE group, but a straightforward interpretation of these results is complicated by the lack of differential fear retention in the control groups. Our results corroborate other failed reconsolidation studies and exemplify the complexity of experimentally investigating this process in humans. Thorough investigation of the interaction between learning and memory reactivation is essential to understand the inconsistencies in the literature and to improve reconsolidation interventions.
Sequential testing enables researchers to monitor and analyze data as it arrives, and decide whether or not to continue data collection depending on the results. Bayesian approaches can mitigate many statistical issues with sequential testing, but we suggest that current discussions of the topic are limited by focusing almost entirely on the mathematical underpinnings of analytic approaches. An important but largely neglected assumption of sequential testing is that the data generating process under investigation remains constant across the experimental cycle. Without care, psychological factors may result in violations of this assumption when sequential testing is used: researchers’ behavior may be changed by the observation of incoming data, in turn influencing the process under investigation. We urge that researchers adopt an ‘insulated’ sequential testing approach, in which research personnel remain blind to the results of interim analyses. We discuss different ways of achieving this, from automation to collaborative inter-lab approaches. As a practical supplement to the issues we raise, we introduce an evolving resource for sequential testing – the Sequential Testing Hub (www.sequentialtesting.com) – including templates and a guide for involving an independent analyst in a sequential testing pipeline, an annotated bibliography centered around the practicalities of sequential analyses, and space for suggestions to help develop this resource further. We aim to show that although unfettered use of sequential testing may raise problems, carefully designed procedures can limit the pitfalls arising from its use, allowing researchers to capitalize on the benefits it provides.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.