Goal-Directed and Habit-Like Modulations of Stimulus Processing during Reinforcement Learning

Abstract

Recent research has shown that perceptual processing of stimuli previously associated with high-value rewards is automatically prioritized even when rewards are no longer available. It has been hypothesized that such reward-related modulation of stimulus salience is conceptually similar to an "attentional habit." Recording event-related potentials in humans during a reinforcement learning task, we show strong evidence in favor of this hypothesis. Resistance to outcome devaluation (the defining feature of a habit) was shown by the stimulus-locked P1 component, reflecting activity in the extrastriate visual cortex. Analysis at longer latencies revealed a positive component (corresponding to the P3b, from 550-700 ms) sensitive to outcome devaluation. Therefore, distinct spatiotemporal patterns of brain activity were observed corresponding to habitual and goal-directed processes. These results demonstrate that reinforcement learning engages both attentional habits and goal-directed processes in parallel. Consequences for brain and computational models of reinforcement learning are discussed. The human attentional network adapts to detect stimuli that predict important rewards. A recent hypothesis suggests that the visual cortex automatically prioritizes reward-related stimuli, driven by cached representations of reward value; that is, stimulus-response habits. Alternatively, the neural system may track the current value of the predicted outcome. Our results demonstrate for the first time that visual cortex activity is increased for reward-related stimuli even when the rewarding event is temporarily devalued. In contrast, longer-latency brain activity was specifically sensitive to transient changes in reward value. Therefore, we show that both habit-like attention and goal-directed processes occur in the same learning episode at different latencies. This result has important consequences for computational models of reinforcement learning.

Avoiding sedentary behaviors requires more cortical resources than avoiding physical activity: An EEG study

“…Because this study was the first to use ERPs to investigate approach and avoidance reactions to physical activity and sedentary behaviors stimuli, it was not possible to formulate specific a priori hypotheses on the spatiotemporal distribution of the potential effects. Therefore, we performed a whole-scalp analysis (64 electrodes) from 0 (stimulus appearance) to 800 ms using a cluster-mass permutation test (Maris & Oostenveld, 2007), which is appropriate for exploratory analyses and delimiting effect boundaries when little guidance is provided by previous research (Groppe, Urbach, & Kutas, 2011;Luque, et al, 2017;Manly, 1997). To fit the analysis with the experimental design and use resampling methods, we perform F-tests of repeated measures ANOVA and the null distribution was computed using permutations of the reduced residuals (Kherad-Pajouh & Renaud, 2015).…”

Motivation and cognitive control in depression

“…At the neural level it has been shown that stimuli related to rewards receive increased sensory processing (Serences, 2008). Studies focusing specifically on attention have shown that the attentional capture by rewarding stimuli can be related to changes in the early processing of such stimuli in the visual cortex (i.e., increase in the P1 component; Donohue et al, 2016;Hickey, Chelazzi, & Theeuwes, 2010;Luque et al, 2017;MacLean & Giesbrecht, 2015). However, other studies have failed to find evidence for such early modulations in the visual cortex, and found changes at later stages of stimulus processing (increased N2pc component and improved decoding in later processing stages; Qi, Zeng, Ding, & Li, 2013;Tankelevitch, Spaak, Rushworth, & Stokes, 2019).…”

Cross-modal integration of reward value during oculomotor planning

“…Among the top-down influences on perception, the effect of reward is particularly important to motivate an agent, facilitate learning, and help the agent to behave adaptively given the limited capacity of both sensory and motor systems. It has been shown that reward acts to modulate selective attention when the subject's performance was directly linked to the monetary reward (Small et al, 2005;Mohanty et al, 2008;Engelmann et al, 2009), even when the monetary reward was no longer task-relevant (Libera and Chelazzi, 2006;Hickey and van Zoest, 2012;Pooresmaeili et al, 2014;Asutay and Västfjäll, 2016;Luque et al, 2017). Rewards may act as guiding signals for learning and optimizing specific attentional operations (Chelazzi et al, 2013a), and not only can reward increase the salience of associated stimuli (Hickey and van Zoest, 2012), but also enhance the suppression of the distractors (Della Libera and Chelazzi, 2009), change the priority maps of space (Chelazzi et al, 2014), and reduce the intrinsic neural noise in the motor and cognitive control (Manohar et al, 2015).…”