Improving the reliability of cognitive task measures: A narrative review

Abstract

Cognitive tasks are capable of providing researchers with crucial insights into the relationship between cognitive processing and psychiatric phenomena across individuals. However, many recent studies have found that task measures exhibit poor reliability, which hampers their utility for individual-differences research. Here we provide a narrative review of approaches to improve the reliability of cognitive task measures. First, we review methods of calculating reliability and discuss some nuances that are specific to cognitive tasks. Then, we introduce a taxonomy of approaches for improving task reliability. Where appropriate, we highlight studies that are exemplary for improving the reliability of specific task measures. We hope that this article can serve as a helpful guide for experimenters who wish to design a new task, or improve an existing one, to achieve sufficient reliability for use in individual-differences research.

The Burden of Reliability: How Measurement Noise Limits Brain-Behaviour Predictions

“…Finally, it is societally unacceptable when resources are spent on acquiring large-scale datasets with unreliable "markers" of behaviour. An optimal choice of targets or increasing reliability via more rigorous testing strategies (Zorowitz & Niv, 2022) is necessary to fully exploit the potential benefits from big data initiatives in neuroscience.…”

“…A wealth of literature discusses ways of improving measurement reliability. Prior to acquisition, this can be achieved by opting for a deeper phenotyping design (Gratton, Nelson, & Gordon, 2022) either in the laboratory or by means of ecological momentary assessment (Moskowitz & Young, 2006), introducing more rigorous testing strategies such as collecting more trials (for an overview see Zorowitz & Niv, 2022), taking measures to increase between-subject variance (Xu et al, 2023) or acquiring multiple assessments for data aggregation (Nikolaidis et al, 2022). In already acquired data, researchers should select relevant measurements with the best psychometric properties.…”

Individual Differences in Computational Psychiatry: A Review of Current Challenges

“…Reliability can be affected by multiple experimental design factors: the number of trials, the duration of inter-stimulus intervals, the number of practice trials, time constrains, overall task difficulty (leading to ceiling or floor effects), the instructions given before the task, or even the population it is tested in, etc. (Henderson et al, 2012;Cooper et al, 2017;McLean et al, 2018;Plummer et al, 2015;Zorowitz and Niv, 2022). This also means that different versions of tasks that are often referred to by the same name (e.g., Go/No-Go task) can have very different reliabilities, depending on the details of implementation.…”

Does the reliability of computational models truly improve with hierarchical modeling? Some recommendations and considerations for the assessment of model parameter reliability

“…The reliability of a parameter estimate is defined as the ratio of true parameter variance to its estimate variance (Zorowitz & Niv, 2022). Assuming that the estimate variance comprises the true parameter variance and error variance, the reliability of a parameter estimate can be defined as…”

“…To improve the reliability of computational models, previous research has focused on improving not only the behavioral measurements (i.e., tasks) but also the parameter estimation methods (Zorowitz & Niv, 2022). In particular, several recent studies have reported that hierarchical Bayesian methods significantly improve the reliability of parameter estimates, compared to conventional maximum likelihood estimation (MLE) (Weidinger et al, 2019;Brown et al, 2020;Waltmann et al, 2022).…”