Credit Assignment in a Motor Decision Making Task Is Influenced by Agency and Not Sensory Prediction Errors

Abstract

Failures to obtain reward can occur from errors in action selection or action execution. Recently, we observed marked differences in choice behavior when the failure to obtain a reward was attributed to errors in action execution compared with errors in action selection (McDougle et al., 2016). Specifically, participants appeared to solve this credit assignment problem by discounting outcomes in which the absence of reward was attributed to errors in action execution. Building on recent evidence indicating relatively direct communication between the cerebellum and basal ganglia, we hypothesized that cerebellar-dependent sensory prediction errors (SPEs), a signal indicating execution failure, could attenuate value updating within a basal ganglia-dependent reinforcement learning system. Here we compared the SPE hypothesis to an alternative, "top-down" hypothesis in which changes in choice behavior reflect participants' sense of agency. In two experiments with male and female human participants, we manipulated the strength of SPEs, along with the participants' sense of agency in the second experiment. The results showed that, whereas the strength of SPE had no effect on choice behavior, participants were much more likely to discount the absence of rewards under conditions in which they believed the reward outcome depended on their ability to produce accurate movements. These results provide strong evidence that SPEs do not directly influence reinforcement learning. Instead, a participant's sense of agency appears to play a significant role in modulating choice behavior when unexpected outcomes can arise from errors in action execution. When learning from the outcome of actions, the brain faces a credit assignment problem: Failures of reward can be attributed to poor choice selection or poor action execution. Here, we test a specific hypothesis that execution errors are implicitly signaled by cerebellar-based sensory prediction errors. We evaluate this hypothesis and compare it with a more "top-down" hypothesis in which the modulation of choice behavior from execution errors reflects participants' sense of agency. We find that sensory prediction errors have no significant effect on reinforcement learning. Instead, instructions influencing participants' belief of causal outcomes appear to be the main factor influencing their choice behavior.

Distinct Neural Signatures of Outcome Monitoring following Selection and Execution Errors

“…Although the majority of neuroscience decision-making research has focused almost exclusively on the former, a number of studies have shown that failed outcomes attributed to sensorimotor errors can markedly alter choice preferences (Green et al, 2010;McDougle et al, 2016;Parvin et al, 2018;McDougle et al, 2019). It has been postulated that these shifts arise from error signals credited to the sensorimotor system attenuating the operation of reinforcement learning processes (McDougle et al, 2016).…”

“…We propose that these findings may be reconciled by considering the potential topdown mechanisms driving execution error processing (Parvin et al, 2018). A recent study demonstrated that a sense of agency, operationalized through the perceived ability to correct for motor errors, biases choice behavior.…”

“…Consistent with this, McDougle et al (2019) have subsequently 54 shown that reward prediction error coding in the human striatum is attenuated following 55 execution versus selection errors. Differences between responses to selection and execution 56 errors have been attributed to a greater sense of "agency" in the latter, with participants' 57 choice biases indicating a belief that they can reduce execution errors by making more 58 accurate movements (Parvin et al, 2018). 59…”

Environmental Consistency Modulation of Error Sensitivity During Motor Adaptation is Explicitly Controlled

“…Previous examinations of consistency effects on sensorimotor learning largely ignored the roles of 408 conscious awareness of the perturbations. Nevertheless, recent evidence illustrated that the sensitivity 409 function of the explicit learning process to different magnitudes of visuomotor rotations is influenced by 410 environmental consistency (Hutter and Taylor, 2018). Interestingly, the observed effect was non-monotonic: sensitivity increased from inconsistent to low-consistent perturbations, but decreased as 412 consistency increased farther.…”

“…Interestingly, the observed effect was non-monotonic: sensitivity increased from inconsistent to low-consistent perturbations, but decreased as 412 consistency increased farther. Importantly, the sensitivity in Hutter and Taylor, 2018, was examined as a 413 function of the perturbation size rather than the experienced visual error, and the distribution of the latter 414 varies at different levels of consistency (Hutter and Taylor, 2018), which may explain the non-415 monotonicity behavior of the sensitivity function. 416…”