Decision theory, reinforcement learning, and the brain

Dayan, Peter; Daw, Nathaniel D.

doi:10.3758/cabn.8.4.429

Cited by 501 publications

(500 citation statements)

References 102 publications

Supporting

Mentioning

486

Contrasting

Order By: Relevance

“…This features set could correspond to neural correlates of decision-making (23,24), probabilistic inference (13,25), and learning and strategy shifts (26,27) that were observed in single-unit recordings in primate and mammalian cortex. Seeking neural correlates of the model presented here would be of particular interest in light of the characterization of the role of memory systems involved in WP (28,29) and other learning and decision-making tasks (30,31), and theoretical models of incremental learning through spike timing-dependent plasticity (32,33).…”

Section: Discussionmentioning

confidence: 99%

High-order feature-based mixture models of classification learning predict individual learning curves and enable personalized teaching

Cohen

Schneidman

2012

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

Pattern classification learning tasks are commonly used to explore learning strategies in human subjects. The universal and individual traits of learning such tasks reflect our cognitive abilities and have been of interest both psychophysically and clinically. From a computational perspective, these tasks are hard, because the number of patterns and rules one could consider even in simple cases is exponentially large. Thus, when we learn to classify we must use simplifying assumptions and generalize. Studies of human behavior in probabilistic learning tasks have focused on rules in which pattern cues are independent, and also described individual behavior in terms of simple, single-cue, feature-based models. Here, we conducted psychophysical experiments in which people learned to classify binary sequences according to deterministic rules of different complexity, including high-order, multicue-dependent rules. We show that human performance on such tasks is very diverse, but that a class of reinforcement learning-like models that use a mixture of features captures individual learning behavior surprisingly well. These models reflect the important role of subjects' priors, and their reliance on high-order features even when learning a low-order rule. Further, we show that these models predict future individual answers to a high degree of accuracy. We then use these models to build personally optimized teaching sessions and boost learning.inference | information | maximum entropy W e regularly learn to classify sensory stimuli into novel categories, often using an impoverished sampling of the stimulus space and the underlying rule: even a "simple" task of classifying patterns of n bits into two categories, requires an implicit mapping of the 2 n possible patterns, which means there are 2 2 n potential deterministic classification rules. It is clear then that when we learn to classify, we cannot simply explore the space of rules and patterns, but instead must rely on simplifying assumptions.Analysis of human learning of deterministic classification rules has focused on modeling of the average behavior of subjects (1-3), and explored the effect of rule complexity on the average level of success (4). Learning to classify according to a probabilistic rule is inherently ambiguous, and so studies of such tasks have focused on simpler rules than those used in deterministic classification. For example, the weather prediction (WP) task (5) requires learning probabilistic associations between multiple cues and a label, where each cue carries independent information about the correct label. Analysis of the learning strategy of individual subjects in this task has compared single-cue or single feature-based strategies, and the possibility of switching between such strategies (6, 7). Associative or Bayesian learning models that rely on simple stimulus features were used to describe the diversity of individual learning dynamics that subjects exhibited and compare between subjects (8), and reflected differences between healthy sub...

show abstract

Section: Discussionmentioning

confidence: 99%

High-order feature-based mixture models of classification learning predict individual learning curves and enable personalized teaching

Cohen

Schneidman

2012

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

show abstract

“…We make the common assumption that critic values are represented in ventral striatum and that phasic signals of dopamine convey the critic prediction error (Dayan & Daw, 2008;Montague et al, 1996;Roesch, Calu, & Schoenbaum, 2007). Actor learning.…”

Section: Opal Model Descriptionmentioning

confidence: 99%

Opponent actor learning (OpAL): Modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive.

Collins¹,

Frank²

2014

Psychological Review

369

455

View full text Add to dashboard Cite

The striatal dopaminergic system has been implicated in reinforcement learning (RL), motor performance, and incentive motivation. Various computational models have been proposed to account for each of these effects individually, but a formal analysis of their interactions is lacking. Here we present a novel algorithmic model expanding the classical actor-critic architecture to include fundamental interactive properties of neural circuit models, incorporating both incentive and learning effects into a single theoretical framework. The standard actor is replaced by a dual opponent actor system representing distinct striatal populations, which come to differentially specialize in discriminating positive and negative action values. Dopamine modulates the degree to which each actor component contributes to both learning and choice discriminations. In contrast to standard frameworks, this model simultaneously captures documented effects of dopamine on both learning and choice incentive-and their interactions-across a variety of studies, including probabilistic RL, effort-based choice, and motor skill learning.Keywords: dopamine, striatum, reinforcement learning, choice incentive, computational model Dopamine plays a crucial role in human and animal cognition, substantially influencing a diversity of processes including reinforcement learning, motivation, incentive, working memory, and effort. Dopaminergic neurons in the substantial nigra and ventral tegmental area project to a very wide set of subcortical and cortical areas, with strongest innervation in the basal ganglia (BG), specifically in the ventral and dorsal striatum. Dysregulation of dopamine is present in a wide array of mental illnesses such as Parkinson's disease, attention-deficit/hyperactivity disorder (ADHD), schizophrenia, and Tourette's syndrome and is a central pharmaceutical target used to treat symptoms across these and numerous other pathologies.Although considerable progress has been made in our understanding of its various distinct roles, there remain fundamental debates concerning its precise mechanisms and functions, especially regarding their integration and interactions. In reward-based decision making in particular, two largely separate traditions have studied the reinforcement learning and the incentive theories of dopamine (Berridge, 2007). Despite solid evidence for both theories, theoretical and empirical studies tend to favor and focus on one or the other interpretation, with little attempt to unify them or to study their interaction. Here we develop an explicit computational analysis of the dual role of striatal dopamine in modulation of incentive motivation (affecting choice), reinforcement learning, and how these processes interact. This endeavor allows us not only to account for both types of findings alone but also those that could not be explained by either theory in isolation. RL Theory of DopamineOne widely accepted theory of dopamine function relates to its role in model-free reinforcement learning (RL). Specifically, phasic f...

show abstract

“…(4) A model of the environment which includes a representation of the environment dynamics required for maximizing the sum of future rewards. These general RL concepts are further explained and elaborated by various authors (Dayan and Daw 2008;Kaelbling et al 1996;Montague et al 2004a;Sutton and Barto 1998;Worgotter and Porr 2005;Dayan and Abbott 2001).…”

Section: Introductionmentioning

confidence: 99%

Computational models of reinforcement learning: the role of dopamine as a reward signal

2010

View full text Add to dashboard Cite

Reinforcement learning is ubiquitous. Unlike other forms of learning, it involves the processing of fast yet content-poor feedback information to correct assumptions about the nature of a task or of a set of stimuli. This feedback information is often delivered as generic rewards or punishments, and has little to do with the stimulus features to be learned. How can such low-content feedback lead to such an efficient learning paradigm? Through a review of existing neuro-computational models of reinforcement learning, we suggest that the efficiency of this type of learning resides in the dynamic and synergistic cooperation of brain systems that use different levels of computations. The implementation of reward signals at the synaptic, cellular, network and system levels give the organism the necessary robustness, adaptability and processing speed required for evolutionary and behavioral success.

show abstract

Decision theory, reinforcement learning, and the brain

Cited by 501 publications

References 102 publications

High-order feature-based mixture models of classification learning predict individual learning curves and enable personalized teaching

High-order feature-based mixture models of classification learning predict individual learning curves and enable personalized teaching

Opponent actor learning (OpAL): Modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive.

Computational models of reinforcement learning: the role of dopamine as a reward signal

Contact Info

Product

Resources

About