Solving the credit assignment problem: explicit and implicit learning of action sequences with probabilistic outcomes

Fu, Wai Tat; Anderson, John R.

doi:10.1007/s00426-007-0113-7

Cited by 48 publications

(38 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There may be both implicit and explicit learning mechanisms that operate in parallel to enable solutions to the credit-assignment problem (Fu and Anderson, 2008), and so these dlPFC neuronal representations may ultimately contribute to one or multiple processes.…”

Section: Discussionmentioning

confidence: 99%

“…The dorsolateral prefrontal cortex (dlPFC), whose neurons have been observed to respond selectively to relevant, attended information (Rainer et al, 1998a;Everling et al, 2002;Lebedev et al, 2004), as well as to maintain information over time (Fuster and Alexander, 1971;Funahashi et al, 1993;Miller et al, 1996;Rainer et al, 1998b;Markowitz et al, 2015), would seem likely to have the basic, necessary properties for solving the structural and temporal credit-assignment problems. Likewise, lateral PFC lesions are typically found to impair performance on tasks that have in common the requirement to use information about choice outcomes to guide future behavior (Dias et al, 1997;Parker and Gaffan, 1998;Mansouri et al, 2007;Rossi et al, 2009;Simmons et al, 2010;Rushworth et al, 2011;Kovach et al, 2012).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Prefrontal Neurons Encode a Solution to the Credit-Assignment Problem

et al. 2017

View full text Add to dashboard Cite

To adapt successfully to our environments, we must use the outcomes of our choices to guide future behavior. Critically, we must be able to correctly assign credit for any particular outcome to the causal features which preceded it. In some cases, the causal features may be immediately evident, whereas in others they may be separated in time or intermingled with irrelevant environmental stimuli, creating a potentially nontrivial credit-assignment problem. We examined the neuronal representation of information relevant for credit assignment in the dorsolateral prefrontal cortex (dlPFC) of two male rhesus macaques performing a task that elicited key aspects of this problem. We found that neurons conveyed the information necessary for credit assignment. Specifically, neuronal activity reflected both the relevant cues and outcomes at the time of feedback and did so in a manner that was stable over time, in contrast to prior reports of representational instability in the dlPFC. Furthermore, these representations were most stable early in learning, when credit assignment was most needed. When the same features were not needed for credit assignment, these neuronal representations were much weaker or absent. These results demonstrate that the activity of dlPFC neurons conforms to the basic requirements of a system that performs credit assignment, and that spiking activity can serve as a stable mechanism that links causes and effects.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Prefrontal Neurons Encode a Solution to the Credit-Assignment Problem

et al. 2017

View full text Add to dashboard Cite

show abstract

“…In two experiments, Fu and Anderson asked participants to make a pair of decisions before receiving feedback (2008a, 2008b). Participants in the dual-task condition performed a memory-intensive n -back task as they made selections.…”

Section: Discussionmentioning

confidence: 99%

Navigating complex decision spaces: Problems and paradigms in sequential choice.

Walsh¹,

Anderson²

2014

Psychological Bulletin

Self Cite

View full text Add to dashboard Cite

To behave adaptively, we must learn from the consequences of our actions. Doing so is difficult when the consequences of an action follow a delay. This introduces the problem of temporal credit assignment. When feedback follows a sequence of decisions, how should the individual assign credit to the intermediate actions that comprise the sequence? Research in reinforcement learning provides two general solutions to this problem: model-free reinforcement learning and model-based reinforcement learning. In this review, we examine connections between stimulus-response and cognitive learning theories, habitual and goal-directed control, and model-free and model-based reinforcement learning. We then consider a range of problems related to temporal credit assignment. These include second-order conditioning and secondary reinforcers, latent learning and detour behavior, partially observable Markov decision processes, actions with distributed outcomes, and hierarchical learning. We ask whether humans and animals, when faced with these problems, behave in a manner consistent with reinforcement learning techniques. Throughout, we seek to identify neural substrates of model-free and model-based reinforcement learning. The former class of techniques is understood in terms of the neurotransmitter dopamine and its effects in the basal ganglia. The latter is understood in terms of a distributed network of regions including the prefrontal cortex, medial temporal lobes cerebellum, and basal ganglia. Not only do reinforcement learning techniques have a natural interpretation in terms of human and animal behavior, but they also provide a useful framework for understanding neural reward valuation and action selection.

show abstract

“…However, Nemeth et al's account would also predict that dual-task conditions should lead to an improvement in procedural learning so a failure to observe an improvement under dual-task conditions would also count against their approach. Evidence shows that striatal-based procedural learning is either improved or no significant differences are observed when participants are given a demanding secondary task (Filoteo, Lauritzen, & Maddox, 2010;Foerde, Knowlton, & Poldrack, 2006;Fu & Anderson, 2008). Thus, whilst the notion that hypnosis results in executive function deficits for suggestible individuals is well-supported, it is possible that this is due to the way in which suggestible individuals actively create dual-task-like conditions.…”

Section: Frontal Executive Functions and Hypnosismentioning

confidence: 99%

The role of frontal executive functions in hypnosis and hypnotic suggestibility.

Parris

2017

Psychology of Consciousness: Theory, Research, and Practice

View full text Add to dashboard Cite

There is both theoretical and empirical evidence supporting a role for frontal executive functions (FEFs) in hypnosis and hypnotic suggestibility. However, the precise nature of this involvement is debated. While there is clear evidence that FEFs are impaired under hypnosis, the cause of this decreased function is unclear. Theories make differing predictions as to the role of FEFs in hypnotic suggestibility, with some arguing that decreased baseline (normal function outside of the hypnotic context) FEFs lead to greater hypnotic suggestibility and others arguing that increased baseline FEFs lead to greater hypnotic suggestibility. Other theories posit that suggestibility is more a consequence of attitude rather than aptitude. The present work provides a critical review of the involvement of FEFs in hypnosis and hypnotic suggestibility. The review considers behavioral evidence from studies employing putative frontal lobe tasks including tests of fluid intelligence and both task- and nontask-based neuroimaging evidence. It is determined that the evidence to date is inconclusive and that more work is needed to establish a necessary and sufficient role for FEFs in hypnosis or hypnotic suggestibility. Recommendations are made for future research.

show abstract

Solving the credit assignment problem: explicit and implicit learning of action sequences with probabilistic outcomes

Cited by 48 publications

References 42 publications

Prefrontal Neurons Encode a Solution to the Credit-Assignment Problem

Prefrontal Neurons Encode a Solution to the Credit-Assignment Problem

Navigating complex decision spaces: Problems and paradigms in sequential choice.

The role of frontal executive functions in hypnosis and hypnotic suggestibility.

Contact Info

Product

Resources

About