May I have your attention, please? Methodological and analytical flexibility in the addiction stroop

Abstract

Background: Theoretical models of addiction predict that an attentional bias toward substance-related cues plays a role in development and maintenance of addictive behaviors, although empirical data testing these predictions are somewhat equivocal. This may in part be a consequence of substantial variability in methods used to operationalize attentional bias. Our aim was to examine the variability in key design and analysis decisions of the addiction Stroop. Method: Using a pre-registered design, we identified 95 studies utilizing an addiction Stroop (46 alcohol, 25 smoking, 24 drug-related). We extracted key information about the design of the Stroop tasks, including; administration (paper-and-pencil vs. computerized), response (key-press vs. voice), design (block vs. mixed). For analysis decisions we extracted information on upper-and lower-bound reaction time cutoffs, removal of data based on standard error cutoffs, removal of participants based on overall performance, type of outcome used, and removal of errors. Results: Based on variability from previous research there are at least 1,451,520 different possible designs of the computerized Alcohol Stroop, 77,760 designs of the computerized Smoking Stroop and 112,640 for the Drug Stroop. Many key design decisions were unreported. Similarly, variability in analyses decisions would allow for 9,000 different methods for analyzing the Alcohol Stroop, 5,376 for the Smoking Stroop and 768 for the Drug Stroop. P-curves suggest data provided evidential value and exploratory meta-regressions suggest that the addiction Stroop effect was not associated with design and analysis decisions. Conclusions: The addiction Stroop effect is seemingly robust, however the adoption of consistent reporting guidelines is necessary to aid reliability and reproducibility.

Raising the bar: improving methodological rigour in cognitive alcohol research

“…functional magnetic resonance imaging [63]), such flexibility is associated with increased false-positive findings, particularly when paired with selective reporting and publication bias [64,65]. Indeed, Jones et al [62] found that key aspects of design and analysis decisions were not disclosed when employing the addiction Stroop task, and our review of the alcohol-attentional bias literature indicates that only one study (1.56%) reported design and analysis decisions a priori through study preregistration.…”

“…Indeed, Jones et al . [62] found that key aspects of design and analysis decisions were not disclosed when employing the addiction Stroop task, and our review of the alcohol‐attentional bias literature indicates that only one study (1.56%) reported design and analysis decisions a priori through study preregistration.…”

“…Similarly, Jones et al . [62] conducted a systematic review of analysis decisions within alcohol and smoking Stroop studies and estimated that more than 7000 analysis pipelines could be attempted. Although these issues extend to the paradigms and techniques employed in other research domains (e.g.…”

“…In addition, they demonstrated that analysing the same data using different cut-off values led to different estimates of internal consistency and test-retest reliability. Similarly, Jones et al [62] conducted a systematic review of analysis decisions within alcohol and smoking Stroop studies and estimated that more than 7000 analysis pipelines could be attempted. Although these issues extend to the paradigms and techniques employed in other research domains (e.g.…”

Test-retest reliability for common tasks in vision science

“…Some discrepancies in reliability scores may also be accounted for by variation in the exact parameters selected for the tasks; the exact specifications of a given task may vary wildly (e.g. Jones, Worrall, Rudin, Duckworth, & Christiansen, 2021 ; Parsons, Kruijt, & Fox, 2019 ). There are also often numerous measures within a task and/or different indices that may be used for performance assessment.…”