Sufficient Reliability of the Behavioral and Computational Read-Outs of a Probabilistic Reversal Learning Task

Abstract

Task-based measures that capture neurocognitive processes can help bridge the gap between brain and behavior. To transfer tasks to clinical application, reliability is a crucial benchmark because it poses an upper bound to potential correlations with other variables (e.g., symptom or brain data). However, the reliability of many task-readouts is low. In this study, we scrutinized the re-test reliability of a probabilistic reversal learning task (PRLT) that is frequently used to characterize cognitive flexibility in psychiatric populations. We analyzed data from N=40 healthy subjects, who completed the PRLT twice. We focused on how individual metrics are derived, i.e., whether data was partially pooled across participants and whether priors were used to inform estimates. We compared the reliability of the resulting indices across sessions, as well as the internal consistency of a selection of indices. We find good to excellent reliability for behavioral indices as derived from mixed-effects models that include data from both sessions. The internal consistency was good to excellent. For indices derived from computational modelling, we find excellent reliability when using hierarchical estimation with empirical priors and including data from both sessions. Our results indicate that the PRLT is well equipped to measure individual differences of cognitive flexibility in reinforcement learning. However, this depends heavily on hierarchical modelling of the longitudinal data (whether sessions are modelled separately or jointly), on estimation methods, and the combination of parameters included in computational models. We discuss implications for the applicability of PRLT indices in psychiatric research and as diagnostic tools.

Consistency within change: Evaluating the psychometric properties of a widely-used predictive-inference task

“…There are multiple factors that may have contributed to these findings. Firstly, these complex measures are interactions between multiple noisy task measures, which is known to lead to a larger overall measurement noise 50,51 . Secondly, we found that multiple model-derived predictors showed a high degree of co-linearity and thus directly affected how well the impact of these metrics could be measured when used in the same model (cf.…”

The Interpretation of Computational Model Parameters Depends on the Context

“…Furthermore, parameter generalizability is naturally bounded by parameter reliability, i.e., the stability of parameter estimates when participants perform the same task twice (test-retest reliability) or when estimating parameters from different subsets of the same dataset (split-half reliability). The reliability of RL models has recently become the focus of several parallel investigations [46, 47, 71, 48], some employing very similar tasks to ours [72]. The investigations collectively suggest that excellent reliability can often be achieved with the right methods, most notably by using hierarchical model fitting.…”

“…Lastly, model parameter reliability might play a crucial role for our results: If parameters lack consistency between two instantiations of the same task (reliability), generalization between different tasks would necessarily be low as well. A recent wave of research, however, has convincingly demonstrated that good reliability is possible for several common RL models [47, 71, 48, 72], and we employ the recommended methods here [61, 53]. In addition, our simulation analysis shows that our approach can detect generalization.…”

“…The investigations collectively suggest that excellent reliability can often be achieved with the right methods, most notably by using hierarchical model fitting. Reliability might still differ between tasks or models, potentially being lower for learning rates than other RL parameters [72], and differing between tasks (e.g., compare [46] to [47]). In this study, we used hierarchical fitting for tasks A and B and assessed a range of qualitative and quantitative measures of model fit for each task [53, 49, 61], boosting our confidence in high reliability of our parameter estimates, and the conclusion that the lack of between-task parameter correlations was not due to a lack of parameter reliability, but a lack of generalizability.…”

Measuring self-regulation in everyday life: reliability and validity of smartphone-based experiments in alcohol use disorder

“…approach in which sessions are modeled jointly. The latter has recently been shown to yield superior reliability estimates in theory and practice in other cognitive tasks (for details, see below; Brown, 2020;Haines, 2021;Waltmann et al, 2021).…”