Reinforcement learning models and their neural correlates: An activation likelihood estimation meta-analysis

Abstract

Reinforcement learning describes motivated behavior in terms of two abstract signals. The representation of discrepancies between expected and actual rewards/punishments – prediction error – is thought to update the expected value of actions and predictive stimuli. Electrophysiological and lesion studies suggest that mesostriatal prediction error signals control behavior through synaptic modification of cortico-striato-thalamic networks. Signals in the ventromedial prefrontal and orbitofrontal cortex are implicated in representing expected value. To obtain unbiased maps of these representations in the human brain, we performed a meta-analysis of functional magnetic resonance imaging studies that employed algorithmic reinforcement learning models, across a variety of experimental paradigms. We found that the ventral striatum (medial and lateral) and midbrain/thalamus represented reward prediction errors, consistent with animal studies. Prediction error signals were also seen in the frontal operculum/insula, particularly for social rewards. In Pavlovian studies, striatal prediction error signals extended into the amygdala, while instrumental tasks engaged the caudate. Prediction error maps were sensitive to the model-fitting procedure (fixed or individually-estimated) and to the extent of spatial smoothing. A correlate of expected value was found in a posterior region of the ventromedial prefrontal cortex, caudal and medial to the orbitofrontal regions identified in animal studies. These findings highlight a reproducible motif of reinforcement learning in the cortico-striatal loops and identify methodological dimensions that may influence the reproducibility of activation patterns across studies.

Behavior, sensitivity, and power of activation likelihood estimation characterized by massive empirical simulation

“…Particularly ALE has been applied to a wide range of different topics from cognitive (Chase et al, 2015; Molenberghs et al, 2012), affective (Kohn et al, 2014; Lamm et al, 2011) and motor (King et al, 2014; Yuan and Brown, 2015) neuroscience to the neurobiology of neurological (Herz et al, 2014; Rehme et al, 2012) and psychiatric (Bludau et al, 2015; Chan et al, 2011; Goodkind et al, 2015) disorders. Given the ever-growing expansion of ALE meta-analyses into smaller research areas in which the eligible literature is not as abundant as for, e.g., working memory or social cognitive tasks, the question regarding the number of experiments that is necessary to perform a valid ALE analyses has become increasingly important over the last years.…”

The involvement of model-based but not model-free learning signals during observational reward learning in the absence of choice

“…Such RPE signals have been extensively reported within striatum during experiential learning in human neuroimaging studies (McClure et al 2003; O’Doherty et al 2003; Gläscher et al 2010). This literature suggests that the ventral aspects of striatum may be involved in encoding RPEs when learning about the value of stimuli as opposed to actions (O’Doherty et al 2004; Cooper et al 2012; Chase et al 2015). In addition, there is growing evidence for the encoding of model-based state-prediction errors (SPEs) by a network of frontoparietal regions (Gläscher et al 2010; Liljeholm et al 2013; Lee et al 2014).…”

Memory-reliant Post-error Slowing Is Associated with Successful Learning and Fronto-occipital Activity

“…On the basis of previous studies (Chase et al, 2015;Garrison et al, 2013;Asaad & Eskandar, 2011), we used an a priori ROI mask of the entire striatum, which was created in WFU PickAtlas (Maldjian, Laurienti, Kraft, & Burdette, 2003), and report small volume corrected (SVC) statistics for this contrast.…”

“…In particular, they have been used to study how responses are adjusted according to the size of the prediction error (Steinberg et al, 2013;Cavanagh, Frank, Klein, & Allen, 2010;Cohen & Ranganath, 2007). One brain area that has consistently been implicated in the coding of positive prediction error is the striatum (see reviews by Chase, Kumar, Eickhoff, & Dombrovski, 2015;Garrison, Erdeniz, & Done, 2013). How the brain codes for negative prediction errors is less apparent.…”