Reliability of Decision-Making and Reinforcement Learning Computational Parameters

Abstract

Background: Computational models can offer mechanistic insight into cognition and therefore have the potential to transform our understanding of psychiatric disorders and their treatment. For translational efforts to be successful, it is imperative that computational measures capture individual characteristics reliably. To date, this issue has received little consideration. Methods: Here we examine the reliability of canonical reinforcement learning and economic models derived from two commonly used tasks. Healthy individuals (N=50) completed a restless four-armed bandit and a calibrated gambling task twice, two weeks apart. Results: Reward and punishment processing parameters from the reinforcement learning model showed fair-to-good reliability, while risk/loss aversion parameters from a prospect theory model exhibited good-to-excellent reliability. Both models were further able to predict future behaviour above chance within individuals. Conclusions: These results suggest that reinforcement learning, and particularly prospect theory measures, represent relatively reliable decision-making mechanisms, which are also unique across individuals, indicating the translational potential of clinically-relevant computational parameters for precision psychiatry.

Individual Differences in Computational Psychiatry: A Review of Current Challenges

“…While these results highlight the benefits of using EB for parameter estimation, the resulting reliabilities are still rather poor. Moreover, many of the studies reporting poor reliabilities are already using EB methods (Moutoussis et al, 2018;Shahar et al, 2019;Brown et al, 2020;Mkrtchian et al, 2021).…”

“…In many cases it is similar to or even lower than the conventional summary statistics measures Shahar et al, 2019;Smith et al, 2021b;Loosen et al, 2022;Waltmann et al, 2022;Hitchcock et al, 2022b). In other cases it offers only a modest improvement over summary statistics (Price et al, 2019;Mkrtchian et al, 2021;Moutoussis et al, 2018;Chung et al, 2017;, and rarely a substantial improvement (Sullivan-Toole et al, 2022;Xu and Stocco, 2021;Smith et al, 2022). Still, some studies achieved better reliability than others and it is important to consider the factors underlying that.…”

“…;Sullivan-Toole et al, 2022). Here we would like to suggest that a more suitable metric of parameter recoverability would be absolute ICC (e.g., see Mkrtchian et al, 2021), which is sensitive to all systematic errors, unlike Pearson or rank correlation.…”

Consistency within change: Evaluating the psychometric properties of a widely-used predictive-inference task

“…That is, the stability of the measure between one task administration and a second administration that follows after a predetermined time interval 35,40 . This is essential when making inferences about stable neurocognitive traits and comparing variability between participants, such as in psychiatric or pharmacological studies 41 . It is therefore crucial to assess both psychometric properties when drawing inferences about intra-and inter-individual differences 35,37,39,42 .…”

Estimating the reliability and stability of cognitive processes contributing to responses on the Implicit Association Test

“…For example, research in functional magnetic resonance imaging (fMRI) has devoted a great deal of research to understanding testretest reliability, as it has significant implications for the clinical applications of fMRI as a tool for diagnosing biomarkers of mental health risk (Bennett & Miller, 2010;Elliott et al, 2020;Herting et al, 2018). In parallel, research on economic decision-making and reinforcement learning has also interrogated the test-retest reliability of computational parameters fit to behavior (Mkrtchian et al, 2021), due to its proposed utility for understanding mental health and psychiatric symptoms. Mirroring this work, MPT parameters can only accurately assess individual differences in implicit attitudes if the parameters can be measured reliably.…”