Plug-and-play supervisory control using muscle and brain signals for real-time gesture and error detection

Abstract

Effective human supervision of robots can be key for ensuring correct robot operation in a variety of potentially safety-critical scenarios. This paper takes a step towards fast and reliable human intervention in supervisory control tasks by combining two streams of human biosignals: muscle and brain activity acquired via EMG and EEG, respectively. It presents continuous classification of left and right hand-gestures using muscle signals, time-locked classification of error-related potentials using brain signals (unconsciously produced when observing an error), and a framework that combines these pipelines to detect and correct robot mistakes during multiple-choice tasks. The resulting hybrid system is evaluated in a "plug-and-play" fashion with 7 untrained subjects supervising an autonomous robot performing a target selection task. Offline analysis further explores the EMG classification performance, and investigates methods to select subsets of training data that may facilitate generalizable plug-and-play classifiers. Keywords Human-robot interaction • EMG control • EEG control • Hybrid control • Gesture detection • Error-related potentials • Plug-and-play supervisory control This is one of the several papers published in Autonomous Robots comprising the Special Issue on Robotics: Science and Systems.

EMG Characterization and Processing in Production Engineering

“…Some examples of the collaborative activities analyzed are supervisory control, fatigue adaptation, hand-over activities, and motion prediction. Artificial intelligence processing techniques are widely used to classify information obtained from EMG acquisition systems, which can be combined with ECG signals [ 54 ] or augmented reality kits [ 114 ]. The analyzed studies are carried out in simulated environments.…”

“…DelPreto et al [ 54 ] use a mixed robotic arm control system to assess hypothetical risk situations: a user monitors the activity of a robotic arm that simulates a drilling operation. A perception of error is identified from the EEG signal, which causes the robot to stop immediately to request assistance.…”

“…Lange et al [ 53 ] develop a mixed system for classifying the movements of grasping and releasing the hand, using EMG and EEG signals, making it possible to integrate both technologies for applications that require the classification of movements. EEG signals could also be used to identify high-level cognitive states, such as error detection in HMI systems [ 54 ].…”

User–Robot Interaction for Safe Navigation of a Quadrotor

“…where K i ∈ R 3 denotes the control gains, and q d defines the desired attitude quaternion representing the user references from (21). More details can be found in ref.…”

“…However, most of these works propose limited solutions to this problem, mainly using out of the shelf products with default settings as in refs. [21][22][23][24][25], which reduce the flexibility of the proposal. Furthermore, safety measures for limiting hazardous gestures are not considered, compromising the safety of operators, specially for novice users.…”