Operant matching is a generic outcome of synaptic plasticity based on the covariance between reward and neural activity

Loewenstein, Yonatan; Seung, H. Sebastian

doi:10.1073/pnas.0505220103

Cited by 97 publications

(138 citation statements)

References 31 publications

(35 reference statements)

Supporting

Mentioning

136

Contrasting

Unclassified

Order By: Relevance

“…Importantly, our framework does not require specialized network structures, such as stimulus-locked tagged delay lines or phase-locked oscillators, used in many previous models (6,18). Our RDE model combines elements of reinforcement learning (19,20), reward modulated plasticity (21)(22)(23)(24) and models of recurrent network dynamics (25)(26)(27)(28). The framework is able to qualitatively account for the most prevalent class of rewardtiming sensitive neurons recorded by Shuler and Bear (9) and can serve, in principle, as a general model of how reward timing can be learned locally in different brain regions.…”

Section: Discussionmentioning

confidence: 99%

“…Demonstrations of reward dependent sustained activity in somatosensory cortex (11) and sustained responses in auditory cortex (8) might indicate that temporal and reward processing occur in lower order areas of the brain than previously thought. Additionally, because local neural populations throughout the cortex meet our model's minimal requirements, the fundamental concept of using an external signal to modulate plasticity (23,24) could be the basis of elementary mechanisms used throughout the brain to process time. Our RDE framework conceptualizes and formalizes how such networks can reliably learn temporal representations and leads to predictions that can be tested experimentally.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Learning reward timing in cortex through reward dependent expression of synaptic plasticity

Gavornik

Shuler

Loewenstein

et al. 2009

Proc. Natl. Acad. Sci. U.S.A.

Self Cite

124

View full text Add to dashboard Cite

The ability to represent time is an essential component of cognition but its neural basis is unknown. Although extensively studied both behaviorally and electrophysiologically, a general theoretical framework describing the elementary neural mechanisms used by the brain to learn temporal representations is lacking. It is commonly believed that the underlying cellular mechanisms reside in high order cortical regions but recent studies show sustained neural activity in primary sensory cortices that can represent the timing of expected reward. Here, we show that local cortical networks can learn temporal representations through a simple framework predicated on reward dependent expression of synaptic plasticity. We assert that temporal representations are stored in the lateral synaptic connections between neurons and demonstrate that reward-modulated plasticity is sufficient to learn these representations. We implement our model numerically to explain reward-time learning in the primary visual cortex (V1), demonstrate experimental support, and suggest additional experimentally verifiable predictions. reinforcment learning ͉ visual cortex O ur brains process time with such instinctual ease that the difficulty of defining what time is, in a neural sense, seems paradoxical. There is a rich literature in experimental neuroscience describing the temporal dynamics of both cellular and system-level neuronal processes and many insightful psychophysical studies have revealed perceptual correlates of time. Despite this, and the clear importance of accurate temporal processing at all levels of behavior, we still know little about how time is represented or used by the brain (1). Temporal processing is classically understood as a higher order function, and although there is some disagreement (2, 3), it is often argued that dedicated structures or regions in the brain are responsible for representing time (4). Because different mechanisms are likely responsible for computing timing at different time scales (1,5,6), and because there is evidence for modality specific temporal mechanisms (7), an alternative possibility is that timing processes develop locally within different brain regions.Recent evidence indicates that temporal representations are expressed in primary sensory cortices (8-10) and that rewardbased reinforcement can affect the form of stimulus driven activity in the primary somatosensory cortex (11-13). In particular, Shuler and Bear (9) showed that neurons in rat primary visual cortex can develop persistent activity, evoked by brief visual stimuli, that robustly represents the temporal interval between a visual stimulus and paired reward (Fig. 1). A mechanistic framework capable of describing how a neural substrate can learn the observed temporal representations does not exist.Here, we explain how these temporal signals can be encoded in recurrent excitatory synaptic connections and how a local network can learn specific temporal instantiations through reward modulated plasticity. Although our model is potentially ...

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Learning reward timing in cortex through reward dependent expression of synaptic plasticity

Gavornik

Shuler

Loewenstein

et al. 2009

Proc. Natl. Acad. Sci. U.S.A.

Self Cite

124

View full text Add to dashboard Cite

show abstract

“…These findings have led to the idea that the traditional Hebbian view of synaptic plasticity driven by pre-and postsynaptic activity has to be augmented by reward prediction error as a third factor [80]. The current tasks of computational neuroscience are to compare the learning rules suggested by the top-down approach with experimental data, and generalize existing concepts in order to evaluate if and under which conditions 3-factor learning rules [82-84] of reward-modulated Hebbian synaptic plasticity can be useful at the macroscopic level of networks [85,86] and behavior [87,88].Reinforcement learning is a textbook example for a synergistic interaction of theory and simulation. Most algorithms are based on mathematical theory, but without an actual implementation and simulation, it is difficult to predict how they perform under realistic conditions.…”

mentioning

confidence: 99%

Theory and Simulation in Neuroscience

Gerstner

Sprekeler

Deco

2012

Science

151

111

View full text Add to dashboard Cite

Abstract:Modeling work in neuroscience can be classified using two different criteria. The first one is the complexity of the model ranging from simplified conceptual models that are amenable to mathematical analysis to detailed models that require simulations in order to understand their properties. The second criterion is that of direction of workflow, which can be from microscopic to macroscopic scales (bottom-up) or from behavioral target functions to properties of components (top-down). We review the interaction of theory and simulation using examples of top-down and bottom-up studies and point to some current developments in the fields of computational and theoretical neuroscience.Mathematical and computational approaches in neuroscience have a long tradition that can be followed back to early mathematical theories of perception [1,2] and of current integration by a neuronal cell membrane [3]. Hodgkin and Huxley combined their experiments with a mathematical description, which they used for simulations on one of the early computers [4]. Hebb's ideas on assembly formation [5] have, already in 1956, inspired simulations on the largest computers available at that time [6]. Since the 1980ies the field of theoretical and computational neuroscience has grown enormously [7].Modern neuroscience methods requiring extensive training have led to a specialization of researchers so that neuroscience today is fragmented into labs working on genes and molecules; on single-cell electrophysiology; on multi-neuron recordings; on cognitive neuroscience and psychophysics, to name just a few. One of the central tasks of computational neuroscience is to bridge these different levels of description by simulation and mathematical theory. The bridge can be built in two different directions. Bottom-up models integrate what is known on a lower level (e.g., properties of ion channels) to explain phenomena observed on a higher level (e.g., generation of action potentials [4,[8][9][10]). Top-down models, on the other hand, start with known cognitive functions of the brain (e.g., working memory), and deduce from these how components (e.g., neurons or groups of neurons) should behave to achieve these functions. Influential examples of the top-down approach are theories of associative memories [11,12], reinforcement learning [13,14], and sparse coding [15,16].Bottom-up and top-down models can either be studied by mathematical theory (theoretical neuroscience) or by computer simulations (computational neuroscience). Theory has the advantage of providing a complete picture of the model behavior for all possible parameter settings, but analytical solutions are restricted to relatively simple models. The aim of theory is therefore to purify biological ideas to the bare minimum, so as to arrive at a 'toy model', which crystallizes a concept in a set of mathematical equations that can be fully understood. Simulations, in contrast, can be applied to all models, simplified as well as complex ones, but they can only sample the model behavior for a...

show abstract

“…Operant behaviour is closely related to the release of DA in the central nervous system (CNS) and the matching paradigm (Herrnstein, 1997), developed within operant psychology, permits detailed experimental analysis of the effects of different rates of reinforcement on choice (Di Chiara, 2002a;Loewenstein and Seung, 2006). 'Matching' refers to the tendency of individual organisms to allocate responses among alternatives in proportion to the reinforcement obtained from each, and is a well-documented phenomenon of both nonhuman and human responding in experimental contexts (Davison and McCarthy, 1988).…”

Section: Rationality and Emotionmentioning

confidence: 99%

Reward, emotion and consumer choice: from neuroeconomics to neurophilosophy

Foxall

2008

J of Consumer Behaviour

View full text Add to dashboard Cite

Neuroeconomics has found no definitive role in the explanation of consumer choice and its undeveloped philosophical basis limits its attempt to explain economic behaviour. The nature of neuroeconomics is explored, especially with respect to what it reveals about the valuation of alternatives, choice and emotion. The tendency of human consumers to discount future rewards illustrates how behavioural and neuroscientific accounts of choice contribute to psychological explanations of choice and the issues this raises for both routine everyday choices and more extreme compulsions. Central to this is the phenomenon of matching in which consumers tend to select the immediately larger or largest reward and the neurophysiological and behavioural bases of this choice. Recognition that rewards are evoked by reinforcement contingencies and that the rewards themselves engender emotional responses via classical conditioning enhances understanding the contribution of neurological activity to the explanation of consumer behaviour. It is argued that neuroeconomics can play a vital explanatory role by providing an evolutionarily consistent warrant for the ascription of intentionality. The Behavioural Perspective Model is used as a template for investigations of consumer choice that lead to iterative theoretical development, forming the basis of a neurophilosophy in which neuroeconomics can find a decisive role.

show abstract

Operant matching is a generic outcome of synaptic plasticity based on the covariance between reward and neural activity

Cited by 97 publications

References 31 publications

Learning reward timing in cortex through reward dependent expression of synaptic plasticity

Learning reward timing in cortex through reward dependent expression of synaptic plasticity

Theory and Simulation in Neuroscience

Reward, emotion and consumer choice: from neuroeconomics to neurophilosophy

Contact Info

Product

Resources

About