The Role of Frontal Cortical and Medial-Temporal Lobe Brain Areas in Learning a Bayesian Prior Belief on Reversals

Abstract

Reversal learning has been extensively studied across species as a task that indexes the ability to flexibly make and reverse deterministic stimulus-reward associations. Although various brain lesions have been found to affect performance on this task, the behavioral processes affected by these lesions have not yet been determined. This task includes at least two kinds of learning. First, subjects have to learn and reverse stimulus-reward associations in each block of trials. Second, subjects become more proficient at reversing choice preferences as they experience more reversals. We have developed a Bayesian approach to separately characterize these two learning processes. Reversal of choice behavior within each block is driven by a combination of evidence that a reversal has occurred, and a prior belief in reversals that evolves with experience across blocks. We applied the approach to behavior obtained from 89 macaques, comprising 12 lesion groups and a control group. We found that animals from all of the groups reversed more quickly as they experienced more reversals, and correspondingly they updated their prior beliefs about reversals at the same rate. However, the initial values of the priors that the various groups of animals brought to the task differed significantly, and it was these initial priors that led to the differences in behavior. Thus, by taking a Bayesian approach we find that variability in reversal-learning performance attributable to different neural systems is primarily driven by different prior beliefs about reversals that each group brings to the task.

The neural basis of reversal learning: An updated perspective

“…This fits with the observation that striatal dopamine transporter binding correlates with individual differences in the use of positive feedback during reversal learning (Stolyarova et al, 2014). Together, these findings suggest dopamine might serve as a neurochemical substrate for the use of Bayesian prior beliefs in reversal learning whereby, as discussed earlier, performance is guided by evidence of reversal occurring from prior experience (Jang et al, 2015). …”

“…This idea is supported by computational modeling (Jang et al, 2015) and in vivo recordings showing that monkey amygdala neurons respond according to the deviation between cached and expected rewards (Belova et al, 2007, Bermudez and Schultz, 2010). This is also consistent with the notion that amygdala encodes a dynamic representation of outcome that incorporates both the stimuli that predict those outcomes, and a history of reward (Paton et al, 2006, Morrison and Salzman, 2010).…”

“…It does so by accumulating a different kind of evidence over multiple reversals to generate a likelihood estimate (known as a prior probability) that a reversal can occur. Using this framework, Costa and colleagues were able to model the subject’s “belief” that a reversal could occur, even before the first reversal, in both standard deterministic (Jang et al, 2015) and probabilistic reversal learning tasks (Costa et al, 2015, Jang et al, 2015). The utilization of such beliefs is consistent with model-based learning, described above.…”

“…Irrespective of their exact design, reversal learning paradigms all likely test the ability to: 1) learn from the rewards received (as well as absence of reward, or reward omission) upon choosing different stimuli (Stalnaker et al, 2015); 2) estimate the likelihood or prior probability that reversals can occur (Costa et al, 2015, Jang et al, 2015); and/or 3) generate an understanding of task or option space (Wilson et al, 2014, Saez et al, 2015) (Figure 3). Defining these different sub-processes in future work will be important, as it also allows more straightforward comparisons between species and paradigms, and the parsing of specific neural systems that contribute to these different aspects of cognitive flexibility.…”

Amygdala and Ventral Striatum Make Distinct Contributions to Reinforcement Learning

“…Although the amygdala exerts Pavlovian control over operant behavior (Cardinal et al, 2002; Robinson et al, 2014; Stuber et al, 2011), it remains unclear if it contributes directly to choice behavior. There is evidence across species both in favor of (Hampton et al, 2007; Rygula et al, 2015; Seymour and Dolan, 2008) and against (Izquierdo et al, 2013; Izquierdo and Murray, 2007; Jang et al, 2015) the amygdala playing a role in RL during choice tasks.…”