Uncertainty alters the balance between incremental learning and episodic memory

Nicholas, Jonathan; Daw, Nathaniel D.; Shohamy, Daphna

doi:10.7554/elife.81679

Cited by 15 publications

(20 citation statements)

References 84 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…how drastically context is shifted to reflect each new sample), the agent can modulate its reliance on temporal abstraction vs constructive, rollout-based simulation, allowing it to balance the costs and benefits of these evaluation regimes depending on circumstances. This is similar to other examples in which, it has been argued, the brain adjusts its decision computations due to similar cost-benefit tradeoffs (Daw, Niv, & Dayan, 2005;Keramati, Dezfouli, & Piray, 2011;Nicholas, Daw, & Shohamy, 2022).…”

Section: Data From Free Recall Experiments Suggest An Intermediate Re...supporting

confidence: 86%

“…Additionally, other manipulations that affect memory, like proactive and retroactive interference, should also have concomitant effects on decisions via enhancement or suppression of particular states and/or outcomes. Conversely, the rationalization of these parameterized memory effects as enabling more efficient choice in various settings suggests that the parameters governing them are potentially malleable, adapting to the statistics of the study material to optimize choice (Nicholas et al, 2022). For instance, when states or study items reflect non-reversible environmental dynamics, a rational RL agent would be expected to dial back the reversibility assumption when learning an SR.…”

Section: Summary Of Findingsmentioning

confidence: 99%

See 1 more Smart Citation

Episodic retrieval for model-based evaluation in sequential decision tasks

Zhou¹,

Talmi²,

Daw

et al. 2023

Preprint

View full text Add to dashboard Cite

It has long been hypothesized that episodic memory supports adaptive decision making by enabling mental simulation of future events. Yet, memory research is often carried out in settings that are far removed from ecological contexts of decision making, and models of adaptive choice, conversely, only invoke episodic memory in highly stylized terms, if it all. To address these gaps, we propose a novel process-level model of choice that grounds model-based evaluation in empirically informed dynamics of episodic recall. In this model, the probability of retrieving each available memory sample is given by the Successor Representation (SR), a biologically plausible world model in reinforcement learning. The evolution of these probabilities based on past retrievals, in turn, is dictated by the Temporal Context Model (TCM), a prominent model of episodic retrieval. Through a series of simulations, we demonstrate that the patterns of episodic retrieval suggested by this model enables flexible computation of decision variables. On this basis, we argue that a number of previously described features of episodic memory serve an adaptive purpose in sequential decision making. For instance, we show that the classic retrieval bias known as contiguity effect, when viewed from a decision making perspective, leads to model-based rollouts for forward simulation. We also show that features of episodic memory such as emotional modulation enable generalization and efficient decisions given limited experience. By bridging theoretical models across these two domains, we make a set of theoretical predictions linking episodic memory properties to adaptive choice in sequential tasks that may guide future empirical endeavors.

show abstract

Section: Data From Free Recall Experiments Suggest An Intermediate Re...supporting

confidence: 86%

Section: Summary Of Findingsmentioning

confidence: 99%

Episodic retrieval for model-based evaluation in sequential decision tasks

Zhou¹,

Talmi²,

Daw

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Finally, recent modeling work has shown that the relative influence of procedural and episodic memory on decision making (see Figure 5b) varies as a function of environmental uncertainty and that this balance may be related to (mal)adaptive choice (Nicholas et al, 2022). As EDs are characterized by greater intolerance of uncertainty (Kesby et al, 2017), similar research in this population may prove meaningful.…”

Section: Moving Forward: Integrating Memory Research In Edsmentioning

confidence: 85%

Back to the future: Progressing memory research in eating disorders

Forester,

Johnson,

Reilly

et al. 2023

Intl J Eating Disorders

View full text Add to dashboard Cite

ObjectiveHuman behaviors, thoughts, and emotions are guided by memories of the past. Thus, there can be little doubt that memory plays a fundamental role in the behaviors (e.g., binging), thoughts (e.g., body‐image concerns), and emotions (e.g., guilt) that characterize eating disorders (EDs). Although a growing body of research has begun to investigate the role of memory in EDs, this literature is limited in numerous ways and has yet to be integrated into an overarching framework.MethodsIn the present article, we provide an operational framework for characterizing different domains of memory, briefly review existing ED memory research within this framework, and highlight crucial gaps in the literature.ResultsWe distinguish between three domains of memory—episodic, procedural, and working—which differ based on functional attributes and underlying neural systems. Most recent ED memory research has focused on procedural memory broadly defined (e.g., reinforcement learning), and findings within all three memory domains are highly mixed. Further, few studies have attempted to assess these different domains simultaneously, though most behavior is achieved through coordination and competition between memory systems. We, therefore, offer recommendations for how to move ED research forward within each domain of memory and how to study the interactions between memory systems, using illustrative examples from other areas of basic and clinical research.DiscussionA stronger and more integrated understanding of the mechanisms that connect memory of past experiences to present ED behavior may yield more comprehensive theoretical models of EDs that guide novel treatment approaches.Public SignificanceMemories of previous eating‐related experiences may contribute to the onset and maintenance of eating disorders (EDs). However, research on the role of memory in EDs is limited, and distinct domains of ED memory research are rarely connected. We, therefore, offer a framework for organizing, progressing, and integrating ED memory research, to provide a better foundation for improving ED treatment and intervention going forward.

show abstract

“…Crucially, each strategy is associated with a different profile of cognitive resource use. Keeping track of individual experiences is much costlier than tracking a single expectation and a confidence interval around it 52,53 and more likely to incur costs when switching between exploring different tables. Prior work suggests individuals switch between using single experiences and summary statistics according to the reliability of each strategy, and the cost of using it 52,53 .…”

Section: Discussionmentioning

confidence: 99%

“…Keeping track of individual experiences is much costlier than tracking a single expectation and a confidence interval around it 52,53 and more likely to incur costs when switching between exploring different tables. Prior work suggests individuals switch between using single experiences and summary statistics according to the reliability of each strategy, and the cost of using it 52,53 . In our case, summary statistics may be perceived as unreliable when overall uncertainty is high, compelling participants to rely on committing individual experiences to working memory 47,50,52,54,55 .…”

Section: Discussionmentioning

confidence: 99%

Human Exploration Strategically Balances Approaching and Avoiding Uncertainty

Abir¹,

Shadlen²,

Shohamy³

2023

Preprint

View full text Add to dashboard Cite

The purpose of exploration is to reduce goal-relevant uncertainty. This can be achieved by choosing to explore the parts of the environment one is most uncertain about. Humans, however, often choose to avoid uncertainty. How do humans balance approaching and avoiding uncertainty during exploration? To answer this question, we developed a task requiring participants to explore a simulated environment towards a clear goal. We compared human choices to the predictions of the optimal exploration policy and a hierarchy of simpler strategies. We found that participants generally explored the object they were more uncertain about. However, when overall uncertainty about choice options was high, participants avoided objects they were more uncertain about, learning instead about better known objects. We examined reaction times and individual differences to understand the costs and benefits of this strategy. We conclude that balancing approaching and avoiding uncertainty ameliorates the costs of exploration in a resource-rational manner.

show abstract

Uncertainty alters the balance between incremental learning and episodic memory

Cited by 15 publications

References 84 publications

Episodic retrieval for model-based evaluation in sequential decision tasks

Episodic retrieval for model-based evaluation in sequential decision tasks

Back to the future: Progressing memory research in eating disorders

Human Exploration Strategically Balances Approaching and Avoiding Uncertainty

Contact Info

Product

Resources

About