Reinforcement learning establishes a minimal metacognitive process to monitor and control motor learning performance

Abstract

Humans and animals develop learning-to-learn strategies throughout their lives to accelerate learning. One theory suggests that this is achieved by a metacognitive process of controlling and monitoring learning. Although such learning-to-learn is also observed in motor learning, the metacognitive aspect of learning regulation has not been considered in classical theories of motor learning. Here, we formulated a minimal mechanism of this process as reinforcement learning of motor learning properties, which regulates a policy for memory update in response to sensory prediction error while monitoring its performance. This theory was confirmed in human motor learning experiments, in which the subjective sense of learning-outcome association determined the direction of up- and down-regulation of both learning speed and memory retention. Thus, it provides a simple, unifying account for variations in learning speeds, where the reinforcement learning mechanism monitors and controls the motor learning process.

Exploring Motor Skill Acquisition in Bimanual Coordination: Insights from Navigating a Novel Maze Task

“…This complexity, even after participants attained a degree of proficiency, notably by the third day of practice, seemed to perpetuate an explorative phase, pushing them to continually probe and experiment with alternative motor strategies 56 . Such findings are supported by ideas in reinforcement learning 57,58 , which suggest that baseline variability increases exploration, which in turn facilitates learning. Interestingly, motor variability has been demonstrated to facilitate learning in supervised error-based learning tasks as well 59 , indicating a broader role for variability in the context of motor learning.…”

The effects of reward and punishment on the performance of ping-pong ball bouncing

“…Importantly, the transfer of reinforcement effects across effectors suggests a regulatory mechanism for motor control that operates at a higher level than the motor learning of individual effectors. This could be explained by the metacognitive process of controlling and monitoring motor learning enabled by the reinforcement learning ( Sugiyama et al, 2023 ). The researchers note that participants regulate their learning and retention rates to maximize reward and minimize punishment.…”

Reward actively engages both implicit and explicit components in dual force field adaptation

“…However, interpretation of a reward-based effect on implicit adaptation is more straightforward. A recent study by Sugiyama and colleagues (2023) [10] has suggested that reward directly influences the learning rate to maximize rewards-a meta-learning approach-which explains variations in implicit adaptation through rewards or punishments. While the exact neural location of such a process is unclear, reward has been shown to affect both basal ganglia [104, 105], which plays a role in both implicit and [106–108] and explicit components [109–111], and cerebellar processing, that has also shown an involvement of both implicit [112–115] and explicit components [78, 116–118].…”

“…On the other hand, in the absence of error information, it would only rely on reward prediction errors to estimate the correct or desired response. Recently it has been suggested that rewards and punishments act at a meta-learning level, directly controlling the learning rates to maximize the reward or minimize the punishment [10]. While reinforcement learning has already been extensively studied in many fields, particularly in both psychology and robotics, fewer studies have investigated its effects in motor adaptation.…”