Reinforcement learning with multiple representations in the basal ganglia loops for sequential motor control

Nakahara, Hiroyuki; Doya, Kenji; Hikosaka, Okihide; Nagano, Saburo

doi:10.1109/ijcnn.1998.686008

Cited by 5 publications

(5 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Based on the two propositions above, we hypothesize that the visual and motor loops, as a whole, achieve both the quick acquisition of novel visuomotor sequences and the robust execution of well-learned sequences Nakahara, 1997;Nakahara et al, 1997). Both loops can work in a parallel manner, and learn sequences concurrently but at different speeds, using reinforcement signals carried by DA projection.…”

Section: Hypothesis On Visual and Motor Loopsmentioning

confidence: 96%

“…The present study investigates how different BG ± TC loops work together in visuomotor sequence control by specifically focusing on the representations used within each loop Nakahara, 1997;Nakahara, Doya, Hikosaka, & Nagano, 1997). We hypothesize that the (dorsolateral) prefrontal and motor loops learn a visuomotor sequence concurrently but at different speeds, using different representations, based on reinforcement signals provided by the DA neurons; further, the cooperation of the two loops, helped by the presupplementary motor area (pre-SMA) that coordinates the contribution of the two loops in a final motor output, achieves both the quick acquisition of novel sequences and the robust execution of well-learned sequences.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Parallel Cortico-Basal Ganglia Mechanisms for Acquisition and Execution of Visuomotor Sequences—A Computational Approach

Nakahara

Doya²,

Hikosaka

2001

Journal of Cognitive Neuroscience

152

110

View full text Add to dashboard Cite

Experimental studies have suggested that many brain areas, including the basal ganglia (BG), contribute to procedural learning. Focusing on the basal ganglia-thalamocortical (BG-TC) system, we propose a computational model to explain how different brain areas work together in procedural learning. The BG-TC system is composed of multiple separate loop circuits. According to our model, two separate BG-TC loops learn a visuomotor sequence concurrently but using different coordinates, one visual, and the other motor. The visual loop includes the dorsolateral prefrontal (DLPF) cortex and the anterior part of the BG, while the motor loop includes the supplementary motor area (SMA) and the posterior BG. The concurrent learning in these loops is based on reinforcement signals carried by dopaminergic (DA) neurons that project divergently to the anterior ("visual") and posterior ("motor") parts of the striatum. It is expected, however, that the visual loop learns a sequence faster than the motor loop due to their different coordinates. The difference in learning speed may lead to inconsistent outputs from the visual and motor loops, and this problem is solved by a mechanism called a "coordinator," which adjusts the contribution of the visual and motor loops to a final motor output. The coordinator is assumed to be in the presupplementary motor area (pre-SMA). We hypothesize that the visual and motor loops, with the help of the coordinator, achieve both the quick acquisition of novel sequences and the robust execution of well-learned sequences. A computational model based on the hypothesis is examined in a series of computer simulations, referring to the results of the 2 x 5 task experiments that have been used on both monkeys and humans. We found that the dual mechanism with the coordinator was superior to the single (visual or motor) mechanism. The model replicated the following essential features of the experimental results: (1) the time course of learning, (2) the effect of opposite hand use, (3) the effect of sequence reversal, and (4) the effects of localized brain inactivations. Our model may account for a common feature of procedural learning: A spatial sequence of discrete actions (subserved by the visual loop) is gradually replaced by a robust motor skill (subserved by the motor loop).

show abstract

Section: Hypothesis On Visual and Motor Loopsmentioning

confidence: 96%

Section: Introductionmentioning

confidence: 99%

Parallel Cortico-Basal Ganglia Mechanisms for Acquisition and Execution of Visuomotor Sequences—A Computational Approach

Nakahara

Doya²,

Hikosaka

2001

Journal of Cognitive Neuroscience

152

110

View full text Add to dashboard Cite

show abstract

“…The relationship between the sequential procedural learning and implicit learning is still unclear. On the basis of these experimental findings, neural network models have been proposed to account for the role of the basal ganglia in learning and execution of sequential procedures (14,24,62,65,81,122,227). A basic anatomical structure common to these models is the BG-TC loop circuit.…”

Section: Relation To Sequential Procedural Learningmentioning

confidence: 99%

Role of the Basal Ganglia in the Control of Purposive Saccadic Eye Movements

Hikosaka

Takikawa

Kawagoe

2000

Physiological Reviews

1,109

900

View full text Add to dashboard Cite

In addition to their well-known role in skeletal movements, the basal ganglia control saccadic eye movements (saccades) by means of their connection to the superior colliculus (SC). The SC receives convergent inputs from cerebral cortical areas and the basal ganglia. To make a saccade to an object purposefully, appropriate signals must be selected out of the cortical inputs, in which the basal ganglia play a crucial role. This is done by the sustained inhibitory input from the substantia nigra pars reticulata (SNr) to the SC. This inhibition can be removed by another inhibition from the caudate nucleus (CD) to the SNr, which results in a disinhibition of the SC. The basal ganglia have another mechanism, involving the external segment of the globus pallidus and the subthalamic nucleus, with which the SNr-SC inhibition can further be enhanced. The sensorimotor signals carried by the basal ganglia neurons are strongly modulated depending on the behavioral context, which reflects working memory, expectation, and attention. Expectation of reward is a critical determinant in that the saccade that has been rewarded is facilitated subsequently. The interaction between cortical and dopaminergic inputs to CD neurons may underlie the behavioral adaptation toward purposeful saccades.

show abstract

“…However, Houk, Adams, and Barto (1995) suggested that action modules resided in the basal ganglia in the form of matrix regions. The learning (both the updating of modules and the updating of the module selection mechanism) is likely to be controlled by processes in the pre-supplementary motor area (Nakahara, Doya, Hikosaka, & Nagano, 1997).…”

Section: Clarion Modelmentioning

confidence: 99%

“…Thus, it may well be the case that the temporal lobe and/or the frontal lobe are both responsible for the explicit processing corresponding to the top level of CLARION. The bottom level of the model, responsible for implicit processing, is likely distributed across a wide range of brain regions including the pre-supplementary motor area, the supplementary motor area, and the motor cortex (Nakahara et al, 1997).…”

Section: Clarion Modelmentioning

confidence: 99%