Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making

Abstract

Classic reinforcement learning (RL) theories cannot explain human behavior in the absence of external reward or when the environment changes. Here, we employ a deep sequential decision-making paradigm with sparse reward and abrupt environmental changes. To explain the behavior of human participants in these environments, we show that RL theories need to include surprise and novelty, each with a distinct role. While novelty drives exploration before the first encounter of a reward, surprise increases the rate of learning of a world-model as well as of model-free action-values. Even though the world-model is available for model-based RL, we find that human decisions are dominated by model-free action choices. The world-model is only marginally used for planning, but it is important to detect surprising events. Our theory predicts human action choices with high probability and allows us to dissociate surprise, novelty, and reward in EEG signals.

Surprise: a unified theory and experimental predictions

“…In a recent study, Xu et al, 2021 showed that the Bayes Factor surprise modulates human reinforcement learning in a volatile multi-step decision-making experiment and correlates with the EEG P300 amplitude at frontal electrodes. They showed that such a correlation is independent of the correlation of the reward prediction error and novelty with the EEG P300 amplitude (Xu et al, 2021). Although these observations support the computation and the use of the Bayes Factor surprise in the brain, the Bayes Factor surprise Most of these previous studies have focused on one measure of surprise and its role and signatures in behavioral and physiological measurements.…”

“…The generative model describes the subjective interpretation of the environment from the point of view of an agent (e.g., a human participant or an animal). Importantly, we assume that the agent takes the possibility into account that the environment may undergo abrupt changes at unknown points in time (similar to Glaze et al, 2015;Heilbron and Meyniel, 2019;Liakoni et al, 2021;Nassar et al, 2010;Xu et al, 2021). Note, however, that we do not assume that the environment has the same dynamics as those assumed by the agent.…”

“…X t and Y t are observable, whereas Θ t and C t are unobservable (hidden). We refer to X t as the cue and to Y t as the observation at time t. Examples of an observation are an image on a computer screen (da Silva & Hare, 2020;Daw et al, 2011;Gläscher et al, 2010;Kolossa et al, 2015;Mars et al, 2008;Xu et al, 2021), an auditory tone (Imada et al, 1993;Lieder et al, 2013;Meyniel et al, 2016;Walz et al, 2015), and an electrical or mechanical stimulation (Esmaeili et al, 2021;Gijsen et al, 2021;Ostwald et al, 2012). The cue variable X t can be interpreted as a predictor of the next observation, since it describes all observable information that is relevant for predicting the observation Y t .…”

“…Here, we apply their definition to our generative model. Similar to Xu et al, 2021, we define the Bayes Factor surprise of observing y t+1 given the cue x t+1 as the ratio of the marginal probability of observing y t+1 given x t+1 and C t+1 = 1 (i.e., assuming a change) to the marginal probability of observing y t+1 given x t+1 and C t+1 = 0 (i.e. assuming no change); formally, we write…”

“…Surprise affects how we predict and perceive our future and how we remember our past (Antony et al, 2021;Dubey & Griffiths, 2020;Gershman et al, 2017;Gottlieb & Oudeyer, 2018;Oudeyer & Kaplan, 2009;Sinclair & Barense, 2018;Soltani & Izquierdo, 2019;Yu & Dayan, 2005). For example, some studies suggest that you would rely less on the weather forecast for your future plans after the snowy morning (Behrens et al, 2007;Findling et al, 2021;Gerstner et al, 2018;Nassar et al, 2010;Soltani & Izquierdo, 2019;Xu et al, 2021;Yu & Dayan, 2005). Other studies predict that you would remember more vividly the face of the random stranger who walked past the street in that very moment you felt surprised (Rouhani & Niv, 2021;Rouhani et al, 2018), and some predict that this moment of surprise might have even modified your memory of another snowy morning in the past (Exton-McGuinness et al, 2015;Gershman et al, 2017;Gershman et al, 2014;Sinclair & Barense, 2018).…”