From Classical Methods to Generative Models: Tackling the Unreliability of Neuroscientific Measures in Mental Health Research

Haines, Nathaniel; Sullivan-Toole, Holly; Olino, Thomas M.

doi:10.1016/j.bpsc.2023.01.001

Cited by 8 publications

(6 citation statements)

References 70 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, we were able to train a classifier that successfully discriminated between high/low confidence trials across testing days and cognitive domains, hence providing strong evidence for test-retest reliability and domain-generality of this neural signature. The testretest reliability of the effect is particularly important given increasing concerns regarding the reliability of scientific findings over the last decade, with numerous replication failures, particularly within neuroscience (Botvinik-Nezer et al, 2019;Haines et al, 2023;Pavlov et al, 2021;Poldrack et al, 2017).…”

Section: Cpp/p3 Is a Reliable And Domain General Predictor Of Confidencementioning

confidence: 99%

“…Though computational models of behaviour, like those often employed to estimate metacognitive performance (Fleming, 2017;Maniscalco & Lau, 2012), offer insights into latent processes, increasing concerns have been raised about their reliability and potential to capture trait-like characteristics (Brown et al, 2020;Hedge et al, 2018;Shahar et al, 2019). Likewise, the reliability of neuroimaging based bio-markers of cognitive functions have also been called into question (Botvinik-Nezer et al, 2019;Botvinik-Nezer & Wager, 2022;Haines et al, 2023;Pavlov et al, 2021;Poldrack et al, 2017).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Two distinct stimulus-locked EEG signatures reliably encode domain-general confidence during decision formation

Kopčanová

Ince

Benwell

2023

Preprint

View full text Add to dashboard Cite

Decision confidence, an internal estimate of how accurate our choices are, is essential for metacognitive self-evaluation and guides behaviour. However, it can be suboptimal and hence understanding the underlying neurocomputational mechanisms is crucial. To do so, it is essential to establish the extent to which both behavioural and neurophysiological measures of metacognition are reliable over time and shared across cognitive domains. The evidence regarding domain-generality of metacognition has been mixed, while the test-retest reliability of the most widely used metacognitive measures has not been reported. Here, in human participants of both sexes, we examined behavioural and electroencephalographic (EEG) measures of metacognition across two tasks that engage distinct cognitive domains - visual perception and semantic memory. The test-retest reliability of all measures was additionally tested across two experimental sessions. The results revealed a dissociation between metacognitive bias and efficiency, whereby only metacognitive bias showed strong test-retest reliability and domain-generality whilst metacognitive efficiency (measured by M-ratio) was neither reliable nor domain-general. Hence, overall confidence calibration (i.e., metacognitive bias) is a stable trait-like characteristic underpinned by domain-general mechanisms whilst metacognitive efficiency may rely on more domain-specific computations. Additionally, we found two distinct stimulus-locked EEG signatures related to the trial-by-trial fluctuations in confidence ratings during decision formation. A late event-related potential was reliably linked to confidence across cognitive domains, while evoked spectral power predicted confidence most reliably in the semantic knowledge domain. Establishing the reliability and domain-generality of neural predictors of confidence represents an important step in advancing our understanding of the mechanisms underlying self-evaluation.

show abstract

Section: Cpp/p3 Is a Reliable And Domain General Predictor Of Confidencementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Two distinct stimulus-locked EEG signatures reliably encode domain-general confidence during decision formation

Kopčanová

Ince

Benwell

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…of (video) game design elements into cognitive tasks, can promote participant engagement [26] and improve the reliability of task measures [27,28]. Moreover, the use of hierarchical Bayesian models -which exert a pooling effect on person-level variables, in effect correcting them for measurement error [29,30] -have been frequently shown to improve the reliability of task measures [31][32][33]. Finally, practice effects can be lessened by designing tasks in such a way that prevents participants from discovering and using task-specific knowledge to enhance their performance on subsequent attempts [34].The aim of the current study was to investigate the reliability and repeatability of a novel version of the Pavlovian go/no-go task.…”

mentioning

confidence: 99%

Improving the Reliability of the Pavlovian Go/No-Go Task

Zorowitz,

Karni,

Paredes

et al. 2023

Preprint

View full text Add to dashboard Cite

Background: The Pavlovian go/no-go task is commonly used to measure individual differences in Pavlovian biases and their interaction with instrumental learning. However, prior research has found suboptimal reliability for computational model-based performance measures for this task, limiting its usefulness in individual-differences research. These studies did not make use of several strategies previously shown to enhance task-measure reliability (e.g., task gamification, hierarchical Bayesian modeling for model estimation). Here we investigated if such approaches could improve the task’s reliability. Methods: Across two experiments, we recruited two independent samples of adult participants (N=103, N=110) to complete a novel, gamified version of the Pavlovian go/no-go task multiple times over several weeks. We used hierarchical Bayesian modeling to derive reinforcement learning model-based indices of participants' task performance, and additionally to estimate the reliability of these measures. Results: In Experiment 1, we observed considerable and unexpected practice effects, with most participants reaching near-ceiling levels of performance with repeat testing. Consequently, the test-retest reliability of some model parameters was unacceptable (range: 0.379–0.973). In Experiment 2, participants completed a modified version of the task designed to lessen these practice effects. We observed greatly reduced practice effects and improved estimates of the test-retest reliability (range: 0.696–0.989). Conclusion: The results demonstrate that model-based measures of performance on the Pavlovian go/no-go task can reach levels of reliability sufficient for use in individual- differences research. However, additional investigation is necessary to validate the modified version of the task in other populations and settings.

show abstract

“…The ecological (or population) fallacy is characterized by the principle that, even if a group in the aggregate is representative of the majority of the individuals within said group, any given individual or subgroup is not necessarily representative of the group at all. Hence, when assumed for the individual, assumptions based on group-level or hierarchical inference are inherently fallacious and invalidate potential conclusions about individual differences, including those applied in computational psychiatry and neurology [323][324][325] for computational phenotyping [29,316,318,319,[326][327][328]. This point is missed in a cognitive-modeling literature now widely and unquestioningly adopting hierarchical Bayesian fitting-a trend motivated by the allure of results that, being biased, merely appear to be cleaner because of unverifiable assumptions about the unknowns of diverse brain states.…”

Section: The Primacy Of Bias and Hysteresis As Well As Individual Dif...mentioning

confidence: 99%

Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts

Colas,

O’Doherty,

Grafton

2024

PLoS Comput Biol

View full text Add to dashboard Cite

Active reinforcement learning enables dynamic prediction and control, where one should not only maximize rewards but also minimize costs such as of inference, decisions, actions, and time. For an embodied agent such as a human, decisions are also shaped by physical aspects of actions. Beyond the effects of reward outcomes on learning processes, to what extent can modeling of behavior in a reinforcement-learning task be complicated by other sources of variance in sequential action choices? What of the effects of action bias (for actions per se) and action hysteresis determined by the history of actions chosen previously? The present study addressed these questions with incremental assembly of models for the sequential choice data from a task with hierarchical structure for additional complexity in learning. With systematic comparison and falsification of computational models, human choices were tested for signatures of parallel modules representing not only an enhanced form of generalized reinforcement learning but also action bias and hysteresis. We found evidence for substantial differences in bias and hysteresis across participants—even comparable in magnitude to the individual differences in learning. Individuals who did not learn well revealed the greatest biases, but those who did learn accurately were also significantly biased. The direction of hysteresis varied among individuals as repetition or, more commonly, alternation biases persisting from multiple previous actions. Considering that these actions were button presses with trivial motor demands, the idiosyncratic forces biasing sequences of action choices were robust enough to suggest ubiquity across individuals and across tasks requiring various actions. In light of how bias and hysteresis function as a heuristic for efficient control that adapts to uncertainty or low motivation by minimizing the cost of effort, these phenomena broaden the consilient theory of a mixture of experts to encompass a mixture of expert and nonexpert controllers of behavior.

show abstract

From Classical Methods to Generative Models: Tackling the Unreliability of Neuroscientific Measures in Mental Health Research

Cited by 8 publications

References 70 publications

Two distinct stimulus-locked EEG signatures reliably encode domain-general confidence during decision formation

Two distinct stimulus-locked EEG signatures reliably encode domain-general confidence during decision formation

Improving the Reliability of the Pavlovian Go/No-Go Task

Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts

Contact Info

Product

Resources

About