Changes in the stimulus‐preceding negativity and lateralized readiness potential during reinforcement learning

Abstract

According to reinforcement learning theory, dopamine-dependent anticipatory processes play a critical role in learning from action outcomes such as feedback or reward. To better understand outcome anticipation, we examined variation in slow cortical potentials and assessed their changes over the course of motor-skill acquisition. Healthy young adults learned a series of precisely timed, key press sequences. Feedback was delivered at a delay of either 2.5 or 8 s, to encourage use of either the striatally mediated, habit learning system or the hippocampus-dependent, episodic memory system, respectively. During the 2.5-s delay, the stimulus-preceding negativity (SPN) was shown to decline in amplitude across trials, confirming previous results from a perceptual categorization task (Morís, Luque, & Rodríguez-Fornells, 2013). This falsifies the hypothesis that SPN reflects specific outcome predictions, on the assumption that the ability to make such predictions should improve as a task is mastered. An SPN was also evident during the 8-s delay, but it increased in amplitude across trials. At the conclusion of the 8-s but not the 2.5-s prefeedback interval, a reversed-polarity lateralized readiness potential (LRP) was noted. It was suggested that this might indicate maintenance of an action representation for comparison with the feedback display. If so, this would constitute the first direct psychophysiological evidence for a popular hypothetical construct in quantitative models of reinforcement learning, the so-called eligibility trace.

Learning to deal with delayed outcomes: EEG oscillatory and slow potentials during the prefeedback interval

“…Figure 6 shows a decline in SPN amplitude as the task was learned which, as in the three previous studies on this topic (Hirao et al., 2017; Morís et al., 2013; Ren et al., 2017), was greatest at sites overlying prefrontal cortex. The trend at posterior sites toward an increase in amplitude across trials was also evident in those three studies.…”

“…Following a similar strategy in this study and the one from 2017 (Ren et al., 2017), we used prefeedback delays that were considerably longer than in prior research (but see Hellwig et al., 2008, and Seidel et al., 2015, for non‐feedback SPN paradigms with extended delays). During the 6‐s delay condition of the present study, we observed a decrease of frontal SPN amplitude across trials, as had Morís et al.…”

“…Following a similar strategy in this study and the one from 2017 (Ren et al, 2017), we used prefeedback delays that were considerably longer than in prior research (but see Hellwig et al, 2008, andSeidel et al, 2015, for non-feedback SPN paradigms with extended delays). During the 6-s delay condition of the present study, we observed a decrease of frontal SPN amplitude across trials, as had Morís et al (2013) with a 1-s delay, Hirao et al (2017) with a 2.5-s delay, and Ren et al (2017) with a 2.5-s delay. With the 6-s delay, though, the amplitude reduction concluded about 2 s prior to feedback onset.…”

“…In the Ren et al. (2017) study, we tested a large number of participants ( N = 60) and adopted a lenient criterion for the minimum number of trials per condition ( N = 10). As noted above, we found a pattern of SPN changes across trials in the condition favoring hippocampus‐dependent learning that differed qualitatively from the typical one.…”

Anterior insula activity and the effect of agency on the Stimulus‐Preceding Negativity

“…For example, an SPN develops prior to task feedback and this negativity is distributed topographically according to stimulus modality—a clear sign of attention (Brunia & van Boxtel, ; Ohgami, Kotani, Hiraku, Aihara, & Ishii, ). Furthermore, SPN amplitude varies directly with the amount of information conveyed by performance feedback (Kotani et al, ), but then declines as the task is mastered and feedback becomes less important (Hirao, Murphy, & Masaki, ; Morís, Luque, & Rodríguez Fornells, ; Ren, Valle‐Inclán, Tukaiev, & Hackley, ).…”