A unified multimodal control framework for human–robot interaction

Abstract

In human-robot interaction, the robot controller must reactively adapt to sudden changes in the environment (due to unpredictable human behaviour). This often requires operating different modes, and managing sudden signal changes from heterogeneous sensor data. In this paper, we present a multimodal sensor-based controller, enabling a robot to adapt to changes in the sensor signals (here, changes in the human collaborator behaviour). Our controller is based on a unified task formalism, and in contrast with classical hybrid visionforce-position control, it enables smooth transitions and weighted combinations of the sensor tasks. The approach is validated in a mock-up industrial scenario, where pose, vision (from both traditional camera and Kinect), and force tasks must be realized either exclusively or simultaneously, for human-robot collaboration.

A collaborative robot for the factory of the future: BAZAR

“…Automatic manipulation of soft materials is a fundamental -yet open -research problem in robotics, despite its numerous applications (e.g., in the food industry). For this, we will profit from our previous works on dual arm control [2], and on merging vision and force for manipulation [14].…”

Symbiotic human-robot collaborative assembly

“…One application of sensor integration is the screwing task proposed by Shauri et al [221], where the trajectory of the robot arm is controlled based on the measurement from the vision system and the robot hand configuration is adjusted based on the pressure/force data. The vision/force integration is also explored in the context of collaborative screw fastening [40], where the data from Kinect, black/white camera and force sensor, deployed to track human hand, screw and contact force, respectively, are used alternately for robot control. De Gea Fernández et al [62] extended sensor data integration from IMU, RGB-D (red, green, blue, depth) camera and laser scanner to robot whole-body control.…”

On Model Adaptation for Sensorimotor Control of Robots

“…for A t = [∂f /∂s t ][∂g/∂x t ] ∈ R m×n as the Jacobian matrix of the system (also known as the interaction matrix in the sensor servoing literature [17]), whose elements depend on the instantaneous configuration x t . The sensorimotor control problem consists in coordinating the motor actions with the feedback signals such that a desired sensory behaviour is achieved.…”