Effects of inference on dopaminergic prediction errors depend on orbitofrontal processing.

Abstract

Dopaminergic reward prediction errors in monkeys reflect inferential reward predictions that well-trained animals can make when associative rules change. Here, in a new analysis of previously described data, we test whether dopaminergic error signals in rats are influenced by inferential predictions and whether such effects depend on the orbitofrontal cortex (OFC). Dopamine neurons were recorded from controls or rats with ipsilateral OFC lesions during performance of a choice task in which odor cues signaled the availability of sucrose reward in two wells. To induce prediction errors, we manipulated either the timing or number of rewards delivered in each well across blocks of trials. Importantly, a change in reward at one well predicted a change in reward at the other on later trials. We compared behavior and neural activity on trials when such inference was possible versus trials involving the same reward change when inference was not possible. Rats responded faster when they could infer an increase in reward compared to when the same reward was coming but they could not infer a change. This inferential prediction was reflected in the firing of dopamine neurons in controls, which changed less to unexpected delivery (or omission) of reward and more to the new high-value cue on inference versus non-inference trials. These effects were absent in dopamine neurons recorded in rats with ipsilateral OFC lesions. Thus, dopaminergic error signals recorded in rats are influenced by both experiential and inferential reward predictions, and the effects of inferential predictions depend on OFC.

Curiosity from the Perspective of Systems Neuroscience

“…This fact indicates that OFC tracks multiple influences on choice, but does not integrate them into a coherent abstract value variable, and suggests that such integration into an abstract value variable, if it occurs, takes place in a downstream structure. (Note that a good deal of evidence supports the idea that the midbrain DA system is downstream of OFC [45,46,47]. More broadly, these results fit into theories about hierarchies in reward processing [48,49,50,51].…”

Predictive Representations in Hippocampal and Prefrontal Hierarchies

“…While the present analyses were focused on spatial navigation, predictive representations are generalizable to non-spatial domains as well. Examples include relational knowledge and category generalization (Constantinescu et al, 2016;Garvert et al, 2017) , abstraction and transfer (Cole et al, 2011) , reward predictions (Takahashi et al, 2017) , associative inference, and schema learning (Hebscher & Gilboa, 2016;McKenzie et al, 2014;Moscovitch & Melo, 1997;Spalding et al, 2018;van Kesteren et al, 2013;Yu, 2018;Zeithamova et al, 2012;Zeithamova & Preston, 2010) . Previous work has proposed a hierarchy of time-scales in the brain (Chen et al, 2015) and indicated a role for hippocampal-prefrontal interactions in integrating episodes to build abstract schema (Schlichting & Preston, 2017) .…”

Learning task-state representations

“…Both electrophysiological [77] and lesion data [5] suggest that even without a functioning OFC, the striatum has access to a state representation that is bound to external stimuli ('observable states'; although unobservable information about timing is also likely computed in the striatum [85]). In contrast, inferred states that incorporate internal information from, e.g., working memory, intended actions, or future goals, and are based on latent-cause inference as discussed above ('partially observable states') seem to require the OFC [5,[86][87]. This can explain why decision making becomes more OFC-dependent as tasks rely more on inference processes necessary to determine the underlying hidden state of a task.…”

“…Error bars: SEM; dashed horizontal: chance baseline; * p<0.05 compared to chance, one-tailed. Figure adapted from [6] and [87].…”