Model-Based Reinforcement Learning Variable Impedance Control for Human-Robot Collaboration

Abstract

Industry 4.0 is taking human-robot collaboration at the center of the production environment. Collaborative robots enhance productivity and flexibility while reducing human's fatigue and the risk of injuries, exploiting advanced control methodologies. However, there is a lack of real-time model-based controllers accounting for the complex human-robot interaction dynamics. With this aim, this paper proposes a Model-Based Reinforcement Learning (MBRL) variable impedance controller to assist human operators in collaborative tasks. More in details, an ensemble of Artificial Neural Networks (ANNs) is used to learn a human-robot interaction dynamic model, capturing uncertainties. Such a learned model is kept updated during collaborative tasks execution. In addition, the learned model is used by a Model Predictive Controller (MPC) with Cross-Entropy Method (CEM). The aim of the MPC+CEM is to online optimize the stiffness and damping impedance control parameters minimizing the human effort (i.e, minimizing the human-robot interaction forces). The proposed approach has been validated through an experimental procedure. A lifting task has been considered as the reference validation application (weight of the manipulated part: 10 kg unknown to the robot controller). A KUKA LBR iiwa 14 R820 has been used as a test platform. Qualitative performance (i.e, questionnaire on perceived collaboration) have been evaluated. Achieved results have been compared with previous developed offline model-free optimized controllers and with the robot manual guidance controller. The proposed MBRL variable impedance controller shows improved humanrobot collaboration. The proposed controller is capable to actively assist the human in the target task, compensating for the unknown part weight. The human-robot interaction dynamic model has been trained with a few initial experiments (30 initial experiments). In addition, the possibility to keep the learning of the human-robot interaction dynamics active allows accounting for the adaptation of human motor system.

An Evaluation Methodology for Interactive Reinforcement Learning with Simulated Users

“…Therefore, we present a methodology for evaluating IntRL agents by employing simulated users that can suitably replicate some characteristics of human interaction. The proposed simulated user methodology can be applied in different contexts such as human-robot collaboration [ 19 , 20 , 21 , 22 ], explainable robotic systems [ 23 , 24 , 25 ], or bioprocess modelling [ 26 ], among others.…”

Semi-Remote Gait Assistance Interface: A Joystick with Visual Feedback Capabilities for Therapists

“…These situations are generally characterized by collaborative work between robots and humans, where safe and efficient physical and cognitive encounters occur [ 47 ]. In particular, where humans and robots interact in complex scenarios where high performance is required [ 48 , 49 ], several strategies have been introduced, such as virtual environments [ 48 ], teleoperation with joysticks [ 50 ], interfaces with virtual impedance [ 50 ], and approaches to force feedback [ 51 ]. Thus, these kinds of methods have, for example, been used to interpret navigation commands and monitor robotic systems such as wheelchairs, exoskeletons, and mobile robots [ 52 , 53 , 54 ] cooperatively.…”

Variable Impedance Control and Learning—A Review

“…However, they do not scale with large datasets and tend to smooth out discontinuities that are typical in interaction tasks. In order to realize sample and computationally efficiency, Roveda et al ( 2020 ) proposed a mode-based RL framework that combines VIC, an ensemble of neural networks to model human–robot interaction dynamics, and an online optimizer of the impedance gains. The ensemble of networks, trained off-line and periodically updated, is exploited to generate a distribution over the predicted interaction that reduces the overfitting and captures uncertainties in the model.…”