Calibration of Cognitive Tests to Address the Reliability Paradox for Decision-Conflict Tasks

Abstract

Standard, well-established cognitive tasks that produce reliable effects in group comparisons also lead to unreliable measurement when assessing individual differences. This “reliability paradox” has been demonstrated in decision-conflict tasks such as the Simon, Flanker, and Stroop tasks, which measure inhibitory capacity. We aimed to address this phenomenon by implementing gamified tasks and combined versions of the standard tests. A series of experiments concluded that a Flanker task and a combined Simon and Stroop task produced reliable estimates of individual differences in under 100 trials per task. In comparison to archival Flanker, Simon, and Stroop data, these two task variations demonstrated both efficient and reliable measurement. Having both theoretical and applied implications regarding how the cognitive testing of individual differences is carried out, our novel approach offers improved estimation of such differences in a way that is both practical and easily implemented.

An app-enhanced cognitive fitness training program for athletes: The rationale and validation protocol

“…The efficacy of the Cognitive Gym protocol will be assessed with (1) pre- and post-intervention cognitive testing administered via CogMission online platform ( Wells et al, 2021 ; Kucina et al, 2022 ), (2) athletes’ self-ratings of their typical cognitive performance over the preceding 3-week period, and (3) coach ratings of athletes’ performance in their sport-specific training to evaluate the all-important far transfer.…”

“…Simon interference is produced when the word appears on the opposite side to the key indicating the color of the word (e.g., pressing the right key for a blue word that is presented on the left). This combination has been shown effective in reducing the testing time and improving the reliability of the resulting individual differences measures ( Wells et al, 2021 ; Kucina et al, 2022 ).…”

“…The CogMission online testing platform (Wells et al, 2021;Kucina et al, 2022) delivers a testing protocol based on a gamified dynamic scenario containing trials made up of the following three tasks, all requiring a binary response by pressing either the Left (z) or Right (/) key on the keyboard: a. Working memory basic task.…”

Improving the reliability of cognitive task measures: A narrative review

“…Regardless, we do not believe that task measures are inherently less reliable than selfreport measures, and that pessimism about task-based individual-differences research is warranted. It is possible to (re)design tasks to achieve good reliability, even to the high levels dictated by conventional standards [21][22][23][24]. The purpose of the current article is to provide a narrative review of approaches to improve task-measure reliability.…”

“…Amplifying the magnitude of an experimental effect (e.g., making a task more challenging, increasing the potency of affect induction) typically increases the range of participants' responses to it. For example, Kucina and colleagues [23] investigated the reliability of cognitive conflict effects in new versions of several standard cognitive-control tasks (e.g., Stroop, Flanker, Simon) that amplified cognitive interference via two task design features. First, they combined multiple sources of cognitive interference in the same task, for example by combining the Stroop and Simon effects to create a "Stroopon" task.…”