Variable Impedance Control - A Reinforcement Learning Approach

Abstract

Abstract-One of the hallmarks of the performance, versatility, and robustness of biological motor control is the ability to adapt the impedance of the overall biomechanical system to different task requirements and stochastic disturbances. A transfer of this principle to robotics is desirable, for instance to enable robots to work robustly and safely in everyday human environments. It is, however, not trivial to derive variable impedance controllers for practical high DOF robotic tasks. In this contribution, we accomplish such gain scheduling with a reinforcement learning approach algorithm, PI 2 (Policy Improvement with Path Integrals). PI 2 is a model-free, sampling based learning method derived from first principles of optimal control. The PI 2 algorithm requires no tuning of algorithmic parameters besides the exploration noise. The designer can thus fully focus on cost function design to specify the task. From the viewpoint of robotics, a particular useful property of PI 2 is that it can scale to problems of many DOFs, so that RL on real robotic systems becomes feasible. We sketch the PI 2 algorithm and its theoretical properties, and how it is applied to gain scheduling. We evaluate our approach by presenting results on two different simulated robotic systems, a 3-DOF Phantom Premium Robot and a 6-DOF Kuka Lightweight Robot. We investigate tasks where the optimal strategy requires both tuning of the impedance of the end-effector, and tuning of a reference trajectory. The results show that we can use path integral based RL not only for planning but also to derive variable gain feedback controllers in realistic scenarios. Thus, the power of variable impedance control is made available to a wide variety of robotic systems and practical applications.

Online learning of varying stiffness through physical human-robot interaction

“…In [4] analytical solutions to optimal control of variable stiffness for maximizing link velocity is reported. Closely related to optimal control is reinforcement learning (RL), which has been used for learning variable impedance policies in [5]. In [6], an EM-based reinforcement learning algorithm initialized by human demonstrations is presented.…”

“…Lines 8-14 computes the the stiffness matrix based on the current window view of Ξ. Then appropriate damping 5 . Refer to Section III for details on the construction of Φ and Ψ.…”

Tendon-Driven Variable Impedance Control Using Reinforcement Learning

“…We introduce a structure by which P I 2 can learn a discontinuous variable impedance control policy that enables tasks requiring contact, motion, and force control during object interaction. With respect to previous results on variable stiffness control with reinforcement learning [3], here we are not using any policy parameterizations that are based on function approximation. Instead, we represent trajectories and control gains as markov diffusion processes.…”

“…More importantly the optimal control in our approach is an average over sampled controls based on how well they performed on the real system and thus no differentiation of the value functions is performed. With respect to previous work on variable stiffness control with reinforcement learning [4,3], our approach does not use function approximators to represent gains or desired trajectories. This lack of policy parameterization allows us to learn non-smooth trajectories and control gains required for tasks that involve contact, which is particularly important when controlling tendonactuated manipulators.…”

“…Tendon-driven systems allow lightweight manipulators which accentuate this problem. Previous work has learned manipulation tasks by learning parameters of smooth trajectory basis functions [19] [3], but the imposed control smoothness limits the dynamic interaction that can occur with the manipulated object.…”