Discovering latent causes in reinforcement learning

Gershman, Samuel J.; Norman, Kenneth A.; Niv, Yaron

doi:10.1016/j.cobeha.2015.07.007

Cited by 130 publications

(135 citation statements)

References 58 publications

Supporting

Mentioning

127

Contrasting

Unclassified

Order By: Relevance

“…These results indicate that prediction error signals in vmPFC and vStr (and perhaps vlPFC and PCC) depend on the current relational structure of the task. The critical difference between the two relational structures in our experiment is not how the prediction error should be computed, but rather how it should be used to inform future behaviour -how should 'credit' for the error be assigned 2,4,[29][30][31][32] One intriguing possibility is therefore that different representations of prediction errors allow different credit assignment for the same prediction errors in the two relational structures. Factorising these computations from the sensory particularities of the task allows them to be rapidly 'remapped' to new stimuli for rapid learning.…”

Section: Vmpfc and Ventral Striatum Represent The Relational Structurmentioning

confidence: 99%

Entorhinal and ventromedial prefrontal cortices abstract and generalise the structure of reinforcement learning problems

Baram

Muller

Nili

et al. 2019

Preprint

View full text Add to dashboard Cite

Knowledge of the structure of a problem, such as relationships between stimuli, enables rapid learning and flexible inference. Humans and other animals can abstract this structural knowledge and generalise it to solve new problems. For example, in spatial reasoning, shortest-path inferences are immediate in new environments. Spatial structural transfer is mediated by grid cells in entorhinal and (in humans) medial prefrontal cortices, which maintain their structure across different environments. Here, using fMRI, we show that entorhinal and mPFC representations perform a much broader role in generalising the structure of problems. We introduce a task-remapping paradigm, where subjects solve multiple reinforcement learning (RL) problems differing in structural or sensory properties. We show that, as with space, entorhinal representations are preserved across different RL problems only if task structure is preserved. In ventromedial PFC, representations of standard RL signals such as prediction error also vary as a function of task structure.

show abstract

Section: Vmpfc and Ventral Striatum Represent The Relational Structurmentioning

confidence: 99%

Entorhinal and ventromedial prefrontal cortices abstract and generalise the structure of reinforcement learning problems

Baram

Muller

Nili

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Rouhani and colleagues [48] further showed that large reward prediction errors boosted performance on multiple memory measures, consistent with the idea that prediction errors led to increased segmentation and reduced interference. By contrast, when changes are small and predictable, a single latent cause is inferred and memories are more malleable and susceptible to interference [49,50]. Neurally, the orbitofrontal cortex, which is sensitive to unobservable states [51], has been shown to represent the probability that each latent cause is currently active during the inference process [52].…”

Section: Inferring Change In the State Of The Worldmentioning

confidence: 99%

Does mental context drift or shift?

DuBrow

Rouhani

Niv

et al. 2017

Current Opinion in Behavioral Sciences

Self Cite

View full text Add to dashboard Cite

Theories of episodic memory have generally proposed that individual memory traces are linked together by a representation of context that drifts slowly over time. Recent data challenge the notion that contextual drift is always slow and passive. In particular, changes in one’s external environment or internal model induce discontinuities in memory that are reflected in sudden changes in neural activity, suggesting that context can shift abruptly. Furthermore, context change effects are sensitive to top-down goals, suggesting that contextual drift may be an active process. These findings call for revising models of the role of context in memory, in order to account for abrupt contextual shifts and the controllable nature of context change.

show abstract

“…This paper advances a positive answer: different functions of context correspond to different latent causal structures (cf. Gershman, Norman, & Niv, 2015), and Bayesian inference over these structures determines which function is appropriate given an animal's training history. This theoretical framework takes a step towards resolving the discrepant experimental data, by providing insight into the factors that govern how context influences learning.…”

Section: Introductionmentioning

confidence: 99%

Context-dependent learning and causal structure

Gershman

2016

Psychon Bull Rev

View full text Add to dashboard Cite

The notion of "context" has played an important but complicated role in animal learning theory. Some studies have found that contextual stimuli (e.g., conditioning chamber) act much like punctate stimuli, entering into competition with other cues as would be predicted by standard associative learning theories. Other studies have found that contextual stimuli act more like "occasion setters," modulating the associative strength of punctate stimuli without themselves acquiring associative strength. Yet other studies have found that context is often largely ignored, resulting in transfer of performance across context changes. This article argues that these diverse functions of context arise in part from different causal interpretations of the environment. A Bayesian theory is presented that infers which causal interpretation best explains an animal's training history, and hence which function of context is appropriate. The theory coherently accounts for a number of disparate experimental results, and quantitatively predicts the results of a new experiment designed to directly test the theory.

show abstract

Discovering latent causes in reinforcement learning

Cited by 130 publications

References 58 publications

Entorhinal and ventromedial prefrontal cortices abstract and generalise the structure of reinforcement learning problems

Entorhinal and ventromedial prefrontal cortices abstract and generalise the structure of reinforcement learning problems

Does mental context drift or shift?

Context-dependent learning and causal structure

Contact Info

Product

Resources

About