Novelty is not Surprise: Human exploratory and adaptive behavior in sequential decision-making

He, Xu; Modirshanechi, Alireza; Lehmann, Marco P.; Gerstner, Wulfram; Herzog, Michael H.

doi:10.1101/2020.09.24.311084

Cited by 8 publications

(41 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The latter approach provides higher flexibility to arbitrate between different policies and is hence more compatible with the rapid change of behavior observed in our human participants that is triggered by the reward value they find in episode 1 (Fig. 5B1-2 and Supplementary Materials); this approach is also more consistent with experimental evidence for partially separate neural pathways of noveltyand reward-induced behaviors 19,65,72–75 . Moreover, our 3rd observation shows that the relative importance of different policies is regulated by the degree of reward optimism – in line with the known influence of environmental variables on the preference for novelty 41,42,74 and curiosity-driven behavior 76,77 .…”

Section: Discussionsupporting

confidence: 83%

The curse of optimism: a persistent distraction by novelty

Modirshanechi

Lin

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Human curiosity has been interpreted as a drive for exploration and modeled by intrinsically motivated reinforcement learning algorithms. An unresolved challenge in machine learning is that these algorithms are prone to distraction by reward-independent stochastic stimuli. We ask whether humans exhibit the same distraction pattern in their behavior as the algorithms. To answer this question, we design a multi-step decision-making paradigm containing an unknown number of states in a stochastic part of the environment, combined with a reward-manipulation that varies optimism of participants regarding the availability of rewards. We show that (i) participants exhibit a persistent distraction by novelty in the stochastic part; (ii) reward optimism increases this distraction; and (iii) the persistent distraction is consistent with the predictions of algorithms driven by novelty but not with ‘optimal’ algorithms driven by information-gain. Our results suggest that humans use sup-optimal but computationally cheap curiosity-driven policies for exploration in complex environments.

show abstract

Section: Discussionsupporting

confidence: 83%

The curse of optimism: a persistent distraction by novelty

Modirshanechi

Lin

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Behavioral responses to novelty have been modeled in different ways across fields. Within the field of reinforcement learning, novelty is often thought of as either a rewarding outcome or a predictor of a potential reward, thereby prompting exploration before the first rewards are received (Kakade and Dayan, 2002; Xu et al, 2021). In this way, it can be incorporated into existing reinforcement learning frameworks.…”

Section: Introductionmentioning

confidence: 99%

“…Recent studies emphasized the computational difference between stimulus novelty and contextual novelty: the former refers to the quality of not being previously experienced or encountered, and the latter refers to the “surprise” when what is experienced does not match with what was expected in time and/or context (i.e. prediction error) (Barto et al, 2013; Kumaran and Maguire, 2007; Ranganath and Rainer, 2003; Xu et al, 2021). Finally, novelty avoidance or neophobia is typically characterized as mal-adaptative behavior but not a sensible action.…”

Section: Introductionmentioning

confidence: 99%

Striatal dopamine explains novelty-induced behavioral dynamics and individual variability in threat prediction

Akiti

Tsutsui-Kimura

Xie

et al. 2021

Preprint

View full text Add to dashboard Cite

Animals exhibit diverse behavioral responses, such as exploration and avoidance, to novel cues in the environment. However, it remains unclear how dopamine neuron-related novelty responses influence behavior. Here, we characterized dynamics of novelty exploration using multi-point tracking (DeepLabCut) and behavioral segmentation (MoSeq). Novelty elicits a characteristic sequence of behavior, starting with investigatory approach and culminating in object engagement or avoidance. Dopamine in the tail of striatum (TS) suppresses engagement, and dopamine responses were predictive of individual variability in behavior. Behavioral dynamics and individual variability were explained by a novel reinforcement learning (RL) model of threat prediction, in which behavior arises from a novelty-induced initial threat prediction (akin to shaping bonus), and a threat prediction that is learned through dopamine-mediated threat prediction errors. These results uncover an algorithmic similarity between reward- and threat-related dopamine sub-systems.

show abstract

“…From a theoretical point of view, the roles of surprise and novelty in the learning process have been studied [Faraji et al, 2018, Xu et al, 2020. Xu et al [2020] found that novelty and surprise play different roles in the learning process: novelty drives exploration while surprise modulates the learning rate. To investigate the impact of surprise, authors have proposed behavioral experiments where the feedback is corrupted, and thus unreliable [Varrier et al, 2019].…”

Section: Discussionmentioning

confidence: 99%

Confidence-controlled Hebbian learning efficiently extracts category membership from stimuli encoded in view of a categorization task

Berlemont

Nadal

2020

Preprint

View full text Add to dashboard Cite

In experiments on perceptual decision-making, individuals learn a categorization task through trial-and-error protocols. We explore the capacity of a decision-making attractor network to learn a categorization task through reward-based, Hebbian type, modifications of the weights incoming from the stimulus encoding layer. For the latter, we assume a standard layer of a large number of stimulus specific neurons. Within the general framework of Hebbian learning, authors have hypothesized that the learning rate is modulated by the reward at each trial. Surprisingly, we find that, when the coding layer has been optimized in view of the categorization task, such reward-modulated Hebbian learning (RMHL) fails to extract efficiently the category membership. In a previous work we showed that the attractor neural networks nonlinear dynamics accounts for behavioral confidence in sequences of decision trials. Taking advantage of these findings, we propose that learning is controlled by confidence, as computed from the neural activity of the decision-making attractor network. Here we show that this confidence-controlled, reward-based, Hebbian learning efficiently extracts categorical information from the optimized coding layer. The proposed learning rule is local, and, in contrast to RMHL, does not require to store the average rewards obtained on previous trials. In addition, we find that the confidence-controlled learning rule achieves near optimal performance.

show abstract

Novelty is not Surprise: Human exploratory and adaptive behavior in sequential decision-making

Cited by 8 publications

References 65 publications

The curse of optimism: a persistent distraction by novelty

The curse of optimism: a persistent distraction by novelty

Striatal dopamine explains novelty-induced behavioral dynamics and individual variability in threat prediction

Confidence-controlled Hebbian learning efficiently extracts category membership from stimuli encoded in view of a categorization task

Contact Info

Product

Resources

About