Biological arm motion through reinforcement learning

Izawa, Jun; Kondo, Toshiyuki; Izutsu, Koji

doi:10.1007/s00422-004-0485-3

Cited by 50 publications

(31 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(14) and Eq. (15). Then, the state of the arm is updated with the Runge-Kutta method and the time changes from t to t + Δt.…”

Section: Discussionmentioning

confidence: 99%

“…Alternatively, the Actor-Critic method which is one of the major frameworks for temporal difference (TD) learning in the Reinforcement Learning [13] is adapted to train the feedback controller in feedback-error-learning. Since it is difficult to imagine that we are born with a trained feedback controller, learning should be performed in a trial-error manner to acquire it [14], [15]. Finally, a FDM is used in the feedback path to overcome sensory delays.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Learning and Control Model of the Arm for Loading

Kim

Kambara

Shin

et al. 2009

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

SUMMARYWe propose a learning and control model of the arm for a loading task in which an object is loaded onto one hand with the other hand, in the sagittal plane. Postural control during object interactions provides important points to motor control theories in terms of how humans handle dynamics changes and use the information of prediction and sensory feedback. For the learning and control model, we coupled a feedback-errorlearning scheme with an Actor-Critic method used as a feedback controller. To overcome sensory delays, a feedforward dynamics model (FDM) was used in the sensory feedback path. We tested the proposed model in simulation using a two-joint arm with six muscles, each with time delays in muscle force generation. By applying the proposed model to the loading task, we showed that motor commands started increasing, before an object was loaded on, to stabilize arm posture. We also found that the FDM contributes to the stabilization by predicting how the hand changes based on contexts of the object and efferent signals. For comparison with other computational models, we present the simulation results of a minimumvariance model.

show abstract

“…(14) and Eq. (15). Then, the state of the arm is updated with the Runge-Kutta method and the time changes from t to t + Δt.…”

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Learning and Control Model of the Arm for Loading

Kim

Kambara

Shin

et al. 2009

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

show abstract

“…The learning problem is simplified by generating exploration noise along two subspaces, one in which the joint stiffness remains constant, and another in which it does not. Most relevant to this article is the fact that the agent learns to increase the impedance through co-contraction when the arm is perturbed with a force of randomly varying orientation [9] or strength [10]. However, applying this approach to higherdimensional systems or real robots might be challenging, as it requires tuning of the time-constant for learning, appropriately setting the bias for the initial stiffness, and determining the appropriate neural network structure.…”

Section: B Variable Impedance Control In Roboticsmentioning

confidence: 99%

“…An early model-free impedance learning approach was presented in [10], where a simulated 2DOF simulated robot arm with antagonistic muscle learns to reach for a target object whilst minimizing motor commands. The reinforcement learning algorithm is implemented as an actor-critic architecture, where the critic learns the value function by minimizing the temporal-difference error, and the actor determines the muscle forces.…”

Section: B Variable Impedance Control In Roboticsmentioning

confidence: 99%

Model-Free Reinforcement Learning of Impedance Control in Stochastic Environments

Stulp

Buchli

Alice

et al. 2012

IEEE Trans. Auton. Mental Dev.

View full text Add to dashboard Cite

Abstract-For humans and robots, variable impedance control is an essential component for ensuring robust and safe physical interaction with the environment. Humans learn to adapt their impedance to specific tasks and environments; a capability which we continually develop and improve until we are well into our twenties. In this article, we reproduce functionally interesting aspects of learning impedance control in humans on a simulated robot platform.As demonstrated in numerous force field tasks, humans combine two strategies to adapt their impedance to perturbations, thereby minimizing position error and energy consumption: 1) if perturbations are unpredictable, subjects increase their impedance through co-contraction; 2) if perturbations are predictable, subjects learn a feed-forward command to offset the perturbation. We show how a 7-DOF simulated robot demonstrates similar behavior with our model-free reinforcement learning algorithm PI 2 , by applying deterministic and stochastic force fields to the robot's end-effector. We show the qualitative similarity between the robot and human movements.Our results provide a biologically plausible approach to learning appropriate impedances purely from experience, without requiring a model of either body or environment dynamics. Not requiring models also facilitates autonomous development for robots, as pre-specified models cannot be provided for each environment a robot might encounter.Index Terms-robots with development and learning skills, using robots to study development and learning, motor system and development, variable impedance control, force field experiments, motion primitives, reinforcement learning

show abstract

“…We adopt the Actor-Critic method [14] in order to acquire a feedback controller for arm reaching. Although we are not the first to apply the Actor-Critic method to reaching tasks, previous models only explain the reaching movement toward one particular target [15]. In our daily life, we are not always reaching to the same target.…”

Section: Introductionmentioning

confidence: 99%

Motor Control-Learning Model for Reaching Movements

Kambara¹,

Kim²,

Shin³

et al. 2006

The 2006 IEEE International Joint Conference on Neural Network Proceedings

View full text Add to dashboard Cite

Abstract-One of the great abilities of the central nervous system (CNS) is that it can learn by itself how to control our body to execute required tasks. Although several motor control models have been proposed to explain well-learned arm reaching movements, those models do not fully consider how the CNS learns to control our body. In this paper, we propose a new motor control model that can learn to generate accurate reaching movements without prior knowledge of arm dynamics. In our model, the control law is learned in a trial-and-error manner using the reward signal. We focus on point-to-point arm reaching task in the sagittal plane and show that accurate reaching movements toward any given point can be learned and generated by our model. Furthermore, the model can predict human subjects' hand trajectories without specifying desired trajectories.

show abstract

Biological arm motion through reinforcement learning

Cited by 50 publications

References 31 publications

Learning and Control Model of the Arm for Loading

Learning and Control Model of the Arm for Loading

Model-Free Reinforcement Learning of Impedance Control in Stochastic Environments

Motor Control-Learning Model for Reaching Movements

Contact Info

Product

Resources

About