Prefrontal Neurons Encode a Solution to the Credit-Assignment Problem

Abstract

To adapt successfully to our environments, we must use the outcomes of our choices to guide future behavior. Critically, we must be able to correctly assign credit for any particular outcome to the causal features which preceded it. In some cases, the causal features may be immediately evident, whereas in others they may be separated in time or intermingled with irrelevant environmental stimuli, creating a potentially nontrivial credit-assignment problem. We examined the neuronal representation of information relevant for credit assignment in the dorsolateral prefrontal cortex (dlPFC) of two male rhesus macaques performing a task that elicited key aspects of this problem. We found that neurons conveyed the information necessary for credit assignment. Specifically, neuronal activity reflected both the relevant cues and outcomes at the time of feedback and did so in a manner that was stable over time, in contrast to prior reports of representational instability in the dlPFC. Furthermore, these representations were most stable early in learning, when credit assignment was most needed. When the same features were not needed for credit assignment, these neuronal representations were much weaker or absent. These results demonstrate that the activity of dlPFC neurons conforms to the basic requirements of a system that performs credit assignment, and that spiking activity can serve as a stable mechanism that links causes and effects.

Population coding of reward prediction errors through opponent organization in the fronto parietal network

“…The ability to integrate information on longer time scales is a consistent feature of frontal and parietal areas, and these areas have been reported to convey across-trial information about context, rules, the presence of conflict as well as rewards (reviewed in 21 ). In most previous studies, however, the extended memories were adaptive for performing a task, correlating with improved rule-based performance or adaptation to conflict 21 , decision making 32,33 and learning 19,20,34 . An additional study reported that dlPFC neurons respond to irrelevant reward history in a complex rule-based task, but found no manifestations of these responses in the monkeys' strategy 35 .…”

“…In frontal cortical areas, however, RPE-like signals have been reported in the ACC, but their specific features are under debate 16,17 . Reward-related activity in the dlPFC has only been reported in complex decision and learning tasks in which expectations and RPEs are difficult to infer (e.g., [18][19][20][21] ). 7A neurons have not been reported to have reward-related activity.…”

A Tutorial for Information Theory in Neuroscience

“…Surrogate data testing or Monte Carlo analysis is frequently the solution to significance testing in information theory analyses (Lindner et al, 2011; Timme et al, 2014b; Wibral et al, 2014a; Asaad et al, 2017). This type of analysis is performed by generating surrogate null model data that preserve certain aspects of the data while randomizing other aspects.…”

“…For instance, research has focused on analyses of electroencephalography (EEG), magnetoencephalography (MEG), and functional MRI (fMRI) data (Jeong et al, 2001; Lizier et al, 2011; Vicente et al, 2011). Research has also focused on trial-based data (Wollstadt et al, 2014; Gomez-Herraro et al, 2015; Asaad et al, 2017) and single-trial time-averaged analyses (Wibral et al, 2013; Timme et al, 2014b, 2016; Nigam et al, 2016). Two particular areas of interest include studies of connectivity (Honey et al, 2007; Ito et al, 2011; Timme et al, 2014b, 2016; Nigam et al, 2016; Wollstadt et al, 2017) and sensory encoding (Bialek et al, 1991; DeWeese and Meister, 1999; Brenner et al, 2000; Panzeri et al, 2001; Butts, 2003; Butts et al, 2007).…”

Microscopic and mesoscopic effects of reward uncertainty in monkey fronto-parietal areas

“…A first question concerns the representation of reward variables, as cells sensitive to rewardexpectation have only been reported in the dlPFC [18][19][20][21] but not in 7A. It is unclear whether this difference reflects a true areal specialization or merely a gap in the empirical literature.…”