Choice Type Impacts Human Reinforcement Learning

Abstract

In reinforcement learning (RL) experiments, participants learn to make rewarding choices in response to different stimuli; RL models use outcomes to estimate stimulus–response values that change incrementally. RL models consider any response type indiscriminately, ranging from more concretely defined motor choices (pressing a key with the index finger), to more general choices that can be executed in a number of ways (selecting dinner at the restaurant). However, does the learning process vary as a function of the choice type? In Experiment 1, we show that it does: Participants were slower and less accurate in learning correct choices of a general format compared with learning more concrete motor actions. Using computational modeling, we show that two mechanisms contribute to this. First, there was evidence of irrelevant credit assignment: The values of motor actions interfered with the values of other choice dimensions, resulting in more incorrect choices when the correct response was not defined by a single motor action; second, information integration for relevant general choices was slower. In Experiment 2, we replicated and further extended the findings from Experiment 1 by showing that slowed learning was attributable to weaker working memory use, rather than slowed RL. In both experiments, we ruled out the explanation that the difference in performance between two condition types was driven by difficulty/different levels of complexity. We conclude that defining a more abstract choice space used by multiple learning systems for credit assignment recruits executive resources, limiting how much such processes then contribute to fast learning.

Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts

“…Regarding state-independent action value Q t (a), this construct is conceptually constrained to align with repetition of rewarded actions. The most obvious interpretation conflates actions with low-level motor output-in contrast to the high-level goals of actions directed toward external stimuli [95,97,[200][201][202][203]-but, under the proper circumstances, there could be cognitive and even strategic aspects to state-independent representations as well for semiexpert control. Sequential action representation under uncertainty can be more abstract than just motor control, such as with action chunking in response to working-memory load [204][205][206].…”

“…Even low-level motor biases, which if disregarding their benefits in lower internal cost might otherwise be considered a disadvantage of embodiment, may also not be so disruptive as part of a tradeoff for which an embodied RL policy has greater potential for robustness in learning per se. Indeed, embodied RL for concrete actions can achieve greater fluency than disembodied RL for symbolic choices abstracted away from motor output [95,99,202]. Benefits of embodied learning may be facilitated by lesser working-memory demands and lesser overall demands from the topology of the action space as a cognitive map [370][371][372][373] more amenable to spatial and embodied representations in the neural circuitry of the basal ganglia and cortex [8,21,22,[374][375][376][377][378].…”

Fundamental processes in sensorimotor learning: Reasoning, refinement, and retrieval

“…This gap in knowledge partly stems from a long-standing neglect of the role of cognition in motor learning because such processes are generally hard to formalize and often exhibit high variability. Conversely, cognitive science frequently overlooks the role of motor control in decision-making; for example, many decisions are constrained by the sensorimotor outcomes associated with making a choice ( Chen et al, 2017 ; McDougle et al, 2016a ; Rmus and Zou, 2022 ). To make progress toward a comprehensive theory of motor learning, one that can explain the intricate cognitive–motor interactions that facilitate successful motor skill acquisition, we propose a ‘3R’ framework that integrates three fundamental concepts shared between the motor learning and cognitive science communities: reasoning, refinement, and retrieval.…”

Fundamental processes in sensorimotor learning: Reasoning, Refinement, and Retrieval

“…This gap in knowledge partly stems from a longstanding neglect of the role of cognition in motor learning because such processes are generally hard to formalize and often exhibit high variability. Conversely, cognitive science frequently overlooks the role of motor control in decision-making; for example, many decisions are constrained by the sensorimotor outcomes associated with making a choice (Chen et al, 2017;McDougle, Boggess, et al, 2016;Rmus et al, 2022). To make progress towards a comprehensive theory of motor learning, one that is capable of explaining the intricate cognitive-motor interactions that facilitate successful motor skill acquisition, we propose a "3R" framework that integrates three fundamental concepts shared between the motor learning and cognitive science communities: Reasoning, Refinement, and Retrieval.…”