Predictive decision making driven by multiple time-linked reward representations in the anterior cingulate cortex

Abstract

In many natural environments the value of a choice gradually gets better or worse as circumstances change. Discerning such trends makes predicting future choice values possible. We show that humans track such trends by comparing estimates of recent and past reward rates, which they are able to hold simultaneously in the dorsal anterior cingulate cortex (dACC). Comparison of recent and past reward rates with positive and negative decision weights is reflected by opposing dACC signals indexing these quantities. The relative strengths of time-linked reward representations in dACC predict whether subjects persist in their current behaviour or switch to an alternative. Computationally, trend-guided choice can be modelled by using a reinforcement-learning mechanism that computes a longer-term estimate (or expectation) of prediction errors. Using such a model, we find a relative predominance of expected prediction errors in dACC, instantaneous prediction errors in the ventral striatum and choice signals in the ventromedial prefrontal cortex.

Excitation and inhibition in anterior cingulate predict use of past experiences

“…Our findings are consistent with an emerging view of dACC in forming, updating and maintaining a model of the world and of behavioral strategies (O'Reilly et al, 2013; Karlsson et al, 2012; Kolling et al, 2014; Wittmann et al, 2016). In our paradigm, it was always advantageous to use information learnt from the outcome of one decision to guide subsequent decisions.…”

A distributed, hierarchical and recurrent framework for reward-based choice

“…Interestingly, the dACC emerges as having the most sustained temporal structure within the regions that have been characterised thus far95. This observation sits well with studies that have linked ACC activity to disengagement from a foreground option in foraging tasks75,76,97,98 and during exploration99. A foreground option may be held in mind over many trials, but disengagments are linked to abrupt and coordinated changes in network activity in the medial PFC100 (FIG.…”

Velocity estimation in reinforcement learning

“…A formal model comparison between the VD model and the hyperbolic extension using PSIS-LOO and WAIC shows that both models have a similar predictive performance. This work accords with other studies that propose humans are sensitive to higher-order variables that control the dynamics of the environment (Behrens et al, 2007;Courville, Daw, & Touretzky, 2006;McGuire, Nassar, Gold, & Kable, 2014;Meder et al, 2017;Ricci & Gallistel, 2017;Wittmann et al, 2016;Yu & Dayan, 2005). In particular, our model suggests that when the environment is changing smoothly at a variable velocity, subjects have an estimate of this quantity and use it to make predictions as suggested in figure 2.…”