Model-based aversive learning in humans is supported by preferential task state reactivation

Abstract

Harm avoidance is critical for survival, yet little is known regarding the underlying neural mechanisms supporting avoidance when we cannot rely on direct trial and error experience. Neural reactivation, and sequential replay, have emerged as potential candidate mechanisms. Here, during an aversive learning task, in conjunction with magnetoencephalography, we show prospective and retrospective reactivation for planning and learning respectively, coupled to evidence for sequential replay. Specifically, when subjects plan in an aversive context, we find preferential reactivation of subsequently chosen goal states and sequential replay of the preceding path. This reactivation was associated with greater hippocampal theta power. At outcome receipt, unchosen goal states are reactivated regardless of outcome valence. However, replay of paths leading to goal states was directionally modulated by outcome valence, with aversive outcomes leading to stronger reverse replay compared to safe outcomes. Our findings suggest that avoidance behaviour involves simulation of alternative future and past outcome states through hippocampally-mediated reactivation and replay.

Neural evidence for the successor representation in choice evaluation

“…By identifying that such behavioral errors are accompanied by neural markers of SR-aligned incorrect state predictions, we provide positive evidence supporting that inference. Conversely, multiple previous studies had identified neural markers of predictive activity across numerous brain areas (Barron et al, 2020;Boorman et al, 2016;Deuker, Bellmund, Navarro Schröder, & Doeller, 2016;Doll et al, 2015;Howard et al, 2015;Stachenfeld et al, 2017;Wimmer & Büchel, 2019;Wise et al, 2021). However, such studies had not provided evidence that such predictive activity relate to behavioral choice evaluation in a manner consistent with SR evaluation.…”

“…The key measure of predictive representations used here was prospective neural activity decoded from sensory cortex. Previous studies have identified that this measure reflects simulated outcomes of choices and relates to between subject variation in succeeding in reward revaluation (Doll et al, 2015;Wise et al, 2021). However, because these studies either required only a single of step state prediction or did not introduce any transition changes, they could not provide evidence distinguishing the key difference between SR and MB representations: multiple-step vs. one-step predictions, respectively.…”

“…On these games both MB and SR make the same predictions, and both predict correct revaluation. Prior studies examining similar manipulations have found greater predictive neural activity aligned with either MB or SR on correct responses, and thus contrast this to MF which predicts incorrect revaluations and no predictive neural activity (Doll et al, 2015;Wise et al, 2021). Although we observed directionally that SR alignment on these games was greater during correct compared to incorrect responses, this difference was not significant (Contrast-estimate: 0.015645, F(1, 98.514) = 0.12128, P = 0.7283).…”

“…Related to this, a limitation of fMRI is that its slow temporal resolution does not allow examination of the temporal dynamics with which the divergent state predictions consistent with either SR or MB occur. However, future work, using faster timescale methods such as MEG (Liu et al, 2021;Wise et al, 2021), or possibly even novel approaches for fMRI (Schuck & Niv, 2019), may be able to further probe the dynamics of state representation, in order directly to test the prediction that any MB-consistent predictions are built up progressively by iterating steps one by one, while SRconsistent predictions are directly retrieved.…”

“…There is suggestive evidence consistent with this predictive representation account, both from general expectancy-related neural activity (Barron et al, 2020;Boorman, Rajendran, O'Reilly, & Behrens, 2016;Bornstein & Daw, 2013;Brunec & Momennejad, 2020;Doll, Duncan, Simon, Shohamy, & Daw, 2015;Garvert, Dolan, & Behrens, 2017;Klein-Flügge, Barron, Brodersen, Dolan, & Behrens, 2013;Schapiro, Rogers, Cordova, Turk-Browne, & Botvinick, 2013;Stachenfeld et al, 2017;Wimmer & Büchel, 2019;Wise, Liu, Chowdhury, & Dolan, 2021) and more specific behavioral slips of action (Momennejad et al, 2017). But as yet, key tests of the mechanism have not been performed.…”