Visual novelty, curiosity, and intrinsic reward in machine learning and the brain

Abstract

A strong preference for novelty emerges in infancy and is prevalent across the animal kingdom. When incorporated into reinforcement-based machine learning algorithms, visual novelty can act as an intrinsic reward signal that vastly increases the efficiency of exploration and expedites learning, particularly in situations where external rewards are difficult to obtain. Here we review parallels between recent developments in novelty-driven machine learning algorithms and our understanding of how visual novelty is computed and signaled in the primate brain. We propose that in the visual system, novelty representations are not configured with the principal goal of detecting novel objects, but rather with the broader goal of flexibly generalizing novelty information across different states in the service of driving novelty-based learning. Highlights• Novelty-based exploration can expedite learning when rewards are sparse • Novelty-based machine learning incorporates novelty into computations of value • In brains and machines, novelty signals are continuous and distributed • Effective novelty-based machine learning requires view and state invariance • IT cortex supports flexible view and state invariant representations of novelty

Discovery of a Role for Rab3b in Habituation and Cocaine Induced Locomotor Activation in Mice Using Heterogeneous Functional Genomic Analysis

“…This is supported by the same locus on Chr 4 being identified when the saline controls were mapped in the original dataset ( Philip et al, 2010 ). A significant literature supports the hypothesis that exposure to novelty and sensory stimuli are reinforcing across species reviewed in Jaegle et al (2019) . Moreover, multiple studies reveal the predictive relationship of novelty seeking and sensation seeking on drug use and effect ( Piazza et al, 1990 ; Belin and Deroche-Gamonet, 2012 ; Dickson et al, 2015 , 2016 ).…”

Novelty is not Surprise: Human exploratory and adaptive behavior in sequential decision-making

“…Humans seek not only explicit rewards such as money or praise [1][2][3][4][5][6][7][8], but also novelty [9,10], an intrinsic reward-like signal which is linked to curiosity [9][10][11][12][13]. In the theory of reinforcement learning, novelty is considered as a drive for exploration [11,[14][15][16], and novelty-driven exploratory actions have been interpreted as steps towards building a model of the world ('world-model') which is then used for action planning [17].…”

Attention in Psychology, Neuroscience, and Machine Learning

“…In biology, novel, confusing, and surprising stimuli can grab attention, and inferotemporal and perirhinal cortex are believed to signal novel visual situations via an adaptation mechanism that reduces responses to familiar inputs. Reinforcement learning algorithms that include novelty as part of the estimate of the value of a state can encourage this kind of exploration (Jaegle et al, 2019). How exactly to calculate surprise or novelty in different circumstances is not always clear, however.…”