Bayesian theories of conditioning in a changing world

Courville, Aaron; Daw, Nathaniel D.; Touretzky, David S.

doi:10.1016/j.tics.2006.05.004

Cited by 419 publications

(429 citation statements)

References 20 publications

Supporting

Mentioning

395

Contrasting

Unclassified

Order By: Relevance

“…Past research on the brain's learning algorithms has primarily focused on how surprising observations, i.e., prediction errors, drive the learning process (22,39). It is only more recently that researchers have realized that the learning rate should also depend on confidence (10,40).…”

Section: Discussionmentioning

confidence: 99%

“…2). A general and normative solution to this problem requires weighting each source of information according to its reliability (3)(4)(5)(6)(7)(8)(9)(10)(11)(12). According to this Bayes-optimal solution, any discrepancy between a new observation and a learned estimate should lead to an update of this internal estimate, but the size of this update should decrease as the prior confidence in this internal estimate increases.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Brain networks for confidence weighting and hierarchical inference during probabilistic learning

Meyniel

Dehaene

2017

Proc. Natl. Acad. Sci. U.S.A.

108

146

View full text Add to dashboard Cite

Learning is difficult when the world fluctuates randomly and ceaselessly. Classical learning algorithms, such as the delta rule with constant learning rate, are not optimal. Mathematically, the optimal learning rule requires weighting prior knowledge and incoming evidence according to their respective reliabilities. This "confidence weighting" implies the maintenance of an accurate estimate of the reliability of what has been learned. Here, using fMRI and an idealobserver analysis, we demonstrate that the brain's learning algorithm relies on confidence weighting. While in the fMRI scanner, human adults attempted to learn the transition probabilities underlying an auditory or visual sequence, and reported their confidence in those estimates. They knew that these transition probabilities could change simultaneously at unpredicted moments, and therefore that the learning problem was inherently hierarchical. Subjective confidence reports tightly followed the predictions derived from the ideal observer. In particular, subjects managed to attach distinct levels of confidence to each learned transition probability, as required by Bayes-optimal inference. Distinct brain areas tracked the likelihood of new observations given current predictions, and the confidence in those predictions. Both signals were combined in the right inferior frontal gyrus, where they operated in agreement with the confidenceweighting model. This brain region also presented signatures of a hierarchical process that disentangles distinct sources of uncertainty. Together, our results provide evidence that the sense of confidence is an essential ingredient of probabilistic learning in the human brain, and that the right inferior frontal gyrus hosts a confidence-based statistical learning algorithm for auditory and visual sequences.T he sensory data that we receive from our environment are often captured by temporal regularities-for instance the colors of traffic lights change according to a predictable greenyellow-red pattern; thunder is often followed by rain, etc. Knowledge of those hidden regularities is often acquired through learning, by aggregating successive observations into summary estimates (e.g., the probability of the light turning red when it is currently yellow). When sensory data are received sequentially, learning can be described as an iterative process that updates the internal estimates each time a new observation is received. Learners must therefore constantly balance two sources of information: their current estimates and the new incoming observations. Any learning algorithm must find a solution to this balancing act. Finding the correct balance is especially critical in a world that is both stochastic and changing (1), i.e., where observations are governed by probabilities that can change over time (a situation called volatility). An excessive reliance on incoming observations will make the learned estimates dominated by fluctuations instead of converging to the true underlying probabilities. Conversely, an excessive reliance o...

show abstract

Section: Discussionmentioning

confidence: 99%

mentioning

confidence: 99%

Brain networks for confidence weighting and hierarchical inference during probabilistic learning

Meyniel

Dehaene

2017

Proc. Natl. Acad. Sci. U.S.A.

108

146

View full text Add to dashboard Cite

show abstract

“…Previous studies have shown that increased uncertainty can lead to higher learning rates (e.g., Behrens, Woolrich, Walton, & Rushworth, 2007;Courville, Daw, & Touretzky, 2006; although see Le Pelley, 2004 andLe Pelly &McLaren, 2003).…”

Section: Discussionmentioning

confidence: 99%

Slower reacquisition after partial extinction in human contingency learning.

Morís¹,

Barberia

Vadillo³

et al. 2017

Journal of Experimental Psychology: Learning, Memory, and Cogni

View full text Add to dashboard Cite

Extinction is a very relevant learning phenomenon from a theoretical and applied point of view. One of its most relevant features is that relapse phenomena often take place once the extinction training has been completed. Accordingly, as extinction-based therapies constitute the most widespread empirically validated treatment of anxiety disorders, one of their most important limitation is this potential relapse. We provide the first demonstration of relapse reduction in human contingency learning using mild aversive stimuli. This effect was found after partial extinction (i.e., reinforced trials were occasionally experienced during extinction, Experiment 1) and progressive extinction treatments (Experiment 3), and it was not only due to differences in uncertainty levels between the partial and a standard extinction group (Experiment 2). The theoretical explanation of these results, the potential uses of this strategy in applied situations, and its current limitations are discussed.Keywords: Extinction, contingency learning, relapse, aversive learning Slower reacquisition after partial extinction 2 Learning relationships between different events relevant to organisms is ubiquitous, and takes place almost constantly (Clark, 2013;Friston, 2003). Most of the time, this acquired knowledge is helpful, as it is the basis of useful predictions about future events or inferences about their relationships. However, it can also be maladaptive. For example, in many anxiety disorders, seemingly harmless cues may become associated to strong and disruptive emotional responses, leading to distress and daily problems for those suffering them (Beckers, Krypotos, Boddez, Effting, & Kindt, 2013;Mineka & Zinbarg, 2006).Fortunately, this type of learning is flexible in the sense that, once it has been acquired, it may be modified or altered. One of the ways in which this modification can take place is through extinction, the repeated presentation of a given cue in the absence of the consequences with which it had been previously associated (Pavlov, 1927). A progressive reduction in the acquired response to this cue is observed after experiencing a series of presentations of the cue alone. This flexibility allows the organism not only to adapt to changes in the cue-outcome relationships as they occur in the environment but also to ease those maladaptive forms of learning that lead to disruptive emotional responses. For example, in cognitive and behavioral therapies of anxiety disorders, exposure therapy -a repeated, systematic, and controlled exposure to the feared cue, in the absence of any traumatic event-seems to be a key component to success (Bouton, 2002, Longmore & Worrell, 2007Norton & Price, 2007). Unfortunately, extinction is not such a robust effect as acquisition itself, and several factors can lead to a recovery of the original fear response, or relapse. Thus, the current challenge is not so much to achieve the fear reduction but to prevent its relapse (Vervliet, Craske, & Hermans, 2013). The return of fear was firs...

show abstract

“…Our experimental design also introduces an immediate change in task demands with the simultaneous presentation of two response options during testing. The resultant uncertainty might constitute a relevant signal for triggering quick adaptive alterations in striatal activity to meet the new task requirements better (27), leading to immediate changes in timed behavior. The effect of the new task demands could be incorporated into ongoing behavior, again through the corticostriatal-thalamic loop, and the modulation of temporal tuning of striatal neurons during this process might be weighted by the differential valuations of the two options [in our case, due to their different probabilities (24)].…”

Section: Discussionmentioning

confidence: 99%

Mice plan decision strategies based on previously learned time intervals, locations, and probabilities

Tosun

Gür

Balcı

2016

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

Animals can shape their timed behaviors based on experienced probabilistic relations in a nearly optimal fashion. On the other hand, it is not clear if they adopt these timed decisions by making computations based on previously learnt task parameters (time intervals, locations, and probabilities) or if they gradually develop their decisions based on trial and error. To address this question, we tested mice in the timed-switching task, which required them to anticipate when (after a short or long delay) and at which of the two delay locations a reward would be presented. The probability of short trials differed between test groups in two experiments. Critically, we first trained mice on relevant task parameters by signaling the active trial with a discriminative stimulus and delivered the corresponding reward after the associated delay without any response requirement (without inducing switching behavior). During the test phase, both options were presented simultaneously to characterize the emergence and temporal characteristics of the switching behavior. Mice exhibited timed-switching behavior starting from the first few test trials, and their performance remained stable throughout testing in the majority of the conditions. Furthermore, as the probability of the short trial increased, mice waited longer before switching from the short to long location (experiment 1). These behavioral adjustments were in directions predicted by reward maximization. These results suggest that rather than gradually adjusting their time-dependent choice behavior, mice abruptly adopted temporal decision strategies by directly integrating their previous knowledge of task parameters into their timed behavior, supporting the model-based representational account of temporal risk assessment.decision making | interval timing | temporal risk assessment | probabilities | mice M any vertebrate species can build temporal expectancies and cluster their anticipatory behaviors around intervals that lead to critical outcomes (1). The resultant timed behaviors have been shown to be sensitive to other crucial elements of environmental statistics, such as the probabilities of different outcomes (2-4). However, how steady-state choice behavior emerges in temporal decision-making tasks that contain probabilistic contingencies remains to be answered. Do animals directly manifest their knowledge of quantities (e.g., time interval, probability) and locations in their timed behavior or do they gradually acquire differential timed response patterns based on reinforcement learning in a fashion stripped of representations? This study aimed to address this fundamental question using a simple temporal decisionmaking task.A class of interval timing paradigms requires the animals to distribute their responses between two (short vs. long latency) options, each of which predicts reward after the corresponding fixed delay. In these cases, the emergent response pattern is first behaviorally investing in the option with a short delay to the reward, and if responding a...

show abstract

Bayesian theories of conditioning in a changing world

Cited by 419 publications

References 20 publications

Brain networks for confidence weighting and hierarchical inference during probabilistic learning

Brain networks for confidence weighting and hierarchical inference during probabilistic learning

Slower reacquisition after partial extinction in human contingency learning.

Mice plan decision strategies based on previously learned time intervals, locations, and probabilities

Contact Info

Product

Resources

About