Revealing Neurocomputational Mechanisms of Reinforcement Learning and Decision-Making With the hBayesDM Package

Abstract

Reinforcement learning and decision-making (RLDM) provide a quantitative framework and computational theories with which we can disentangle psychiatric conditions into the basic dimensions of neurocognitive functioning. RLDM offer a novel approach to assessing and potentially diagnosing psychiatric patients, and there is growing enthusiasm for both RLDM and computational psychiatry among clinical researchers. Such a framework can also provide insights into the brain substrates of particular RLDM processes, as exemplified by model-based analysis of data from functional magnetic resonance imaging (fMRI) or electroencephalography (EEG). However, researchers often find the approach too technical and have difficulty adopting it for their research. Thus, a critical need remains to develop a user-friendly tool for the wide dissemination of computational psychiatric methods. We introduce an R package called hBayesDM (hierarchical Bayesian modeling of Decision-Making tasks), which offers computational modeling of an array of RLDM tasks and social exchange games. The hBayesDM package offers state-of-the-art hierarchical Bayesian modeling, in which both individual and group parameters (i.e., posterior distributions) are estimated simultaneously in a mutually constraining fashion. At the same time, the package is extremely user-friendly: users can perform computational modeling, output visualization, and Bayesian model comparisons, each with a single line of coding. Users can also extract the trial-by-trial latent variables (e.g., prediction errors) required for model-based fMRI/EEG. With the hBayesDM package, we anticipate that anyone with minimal knowledge of programming can take advantage of cutting-edge computational-modeling approaches to investigate the underlying processes of and interactions between multiple decision-making (e.g., goal-directed, habitual, and Pavlovian) systems. In this way, we expect that the hBayesDM package will contribute to the dissemination of advanced modeling approaches and enable a wide range of researchers to easily perform computational psychiatric research within different populations.

Selective Effects of the Loss of NMDA or mGluR5 Receptors in the Reward System on Adaptive Decision-Making

“…To perform HBA, we used the hBayesDM package (Ahn et al, 2017), which is an R package that offers HBA of various computational models and tasks using the Stan software (Carpenter et al, 2017). The hBayesDM functions of models 1–3 are prl_rp , prl_fictitious_woa , and prl_fictitious_rp_woa , respectively.…”

Rapid, precise, and reliable measurement of delay discounting using a Bayesian learning algorithm

“…Where and are subjective values of the SS and LL options. To estimate the two parameters of the hyperbolic model in the SC method, we used the hBayesDM package 53 automatically estimated on each trial. Note that estimation of discounting rate (k) was of primary interest in this project.…”

“…Where 66 and 77 are subjective values of the SS and LL options. To estimate the two parameters of the hyperbolic model in the staircase method, we used the hBayesDM package (Ahn, Haines, & Zhang, 2017). The hBayesDM package (https://github.com/CCS-Lab/hBayesDM) offers hierarchical and non-hierarchical Bayesian analysis of various computational models and tasks using the Stan software (Carpenter et al, 2016).…”

Decomposing Loss Aversion from Gaze Allocation and Pupil Dilation

“…was then employed to identify the values of λ and μ . The estimation was implemented using a hierarchical Bayesian approach (hBayesDM package in R) (45).…”