Model-free and model-based reward prediction errors in EEG

Abstract

Learning theorists posit two reinforcement learning systems: model-free and model-based. Model-based learning incorporates knowledge about structure and contingencies in the world to assign candidate actions with an expected value. Model-free learning is ignorant of the world's structure; instead, actions hold a value based on prior reinforcement, with this value updated by expectancy violation in the form of a reward prediction error. Because they use such different learning mechanisms, it has been previously assumed that model-based and model-free learning are computationally dissociated in the brain. However, recent fMRI evidence suggests that the brain may compute reward prediction errors to both model-free and model-based estimates of value, signalling the possibility that these systems interact. Because of its poor temporal resolution, fMRI risks confounding reward prediction errors with other feedback-related neural activity. In the present study, EEG was used to show the presence of both model-based and model-free reward prediction errors and their place in a temporal sequence of events including state prediction errors and action value updates. This demonstration of model-based prediction errors questions a long-held assumption that model-free and model-based learning are dissociated in the brain.

Model-based learning deficits in compulsivity are linked to faulty representations of task structure

“…Previous EEG studies of the two-step task (Eppinger et al, 2017;Sambrook et al, 2018;Shahnazian et al, 2019) showed that the P300 was associated with state transitions. However, the inconsistent direction on the effects raises doubt as to how these differences should be interpreted.…”

“…The P300 or P3b has well-established sensitivity to stimulus probability, exhibiting larger peak amplitudes for less probable stimuli (Polich & Margala, 1997). Prior research in healthy humans thus hypothesised that the P300 may be a marker of sensitivity to state transitions on the two-step task, though these studies have yielded inconsistent results, with some finding greater P300 amplitudes for rare versus common transitions (Sambrook, Hardwick, Wills, & Goslin, 2018;Shahnazian, Ribas-Fernandes, & Holroyd, 2019) and one finding the opposite (Eppinger, Walter, & Li, 2017). Here, we examined the second stage stimulus-locked P300 and found a significant main effect of transition type (β = 0.15, SE = 0.07, p = 0.03), consistent with Sambrook et al (2018) and Shahnazian et al (2019) whereby greater P300 amplitude was observed after rare versus common transitions (Figure 3).…”

“…Prior research in healthy humans thus hypothesised that the P300 may be a marker of sensitivity to state transitions on the two-step task, though these studies have yielded inconsistent results, with some finding greater P300 amplitudes for rare versus common transitions (Sambrook, Hardwick, Wills, & Goslin, 2018;Shahnazian, Ribas-Fernandes, & Holroyd, 2019) and one finding the opposite (Eppinger, Walter, & Li, 2017). Here, we examined the second stage stimulus-locked P300 and found a significant main effect of transition type (β = 0.15, SE = 0.07, p = 0.03), consistent with Sambrook et al (2018) and Shahnazian et al (2019) whereby greater P300 amplitude was observed after rare versus common transitions (Figure 3). However, this differential rare versus common signal was not larger in individuals high in model-based planning (β = 0.07, SE = 0.08, p = 0.35), nor did it show any association to compulsivity (β = 0.09, SE = 0.08, p = 0.24).…”

Electrophysiological correlates of state transition prediction errors

“…However, in contrast to Experiment 1, because this experiment involved rewards, it allowed for a possible confound between SPEs and RPEs. Note that although the concurrent effects of RPEs and SPEs can be disentangled using advanced modeling approaches (see Sambrook, Hardwick, Wills, & Goslin, 2018), such approaches require the application of a linear regression on multiple variables (in the present case, SPE, RPE and the ongoing EEG), and so have relatively limited statistical power.…”

The influence of internal models on feedback-related brain activity

“…It is assumed that these two sources of information—past experience and internal models—improve decision making via two different learning mechanisms called model-free learning and model-based learning (Daw & O’Doherty, 2014 ; Dayan & Berridge, 2014 ; O’Doherty, Cockburn, & Pauli, 2017 ). Despite the long held assumption of computationally dissociable learning mechanisms, recent literature suggests an integration of model-free and model-based information at the level of feedback processing (Daw, Gershman, Seymour, Dayan, & Dolan, 2011 ; Sambrook, Hardwick, Wills, & Goslin, 2018 ), and this integration might be sensitive to the structure of the environment as represented by an internal model (Eppinger, Walter, & Li, 2017 ; Lee, Shimojo, & O’Doherty, 2014 ). We ask how different environmental structures and thus internal models exert influence on both behavioral and neural aspects of feedback evaluation.…”