Multimodal execution monitoring for anomaly detection during robot manipulation

Abstract

Online detection of anomalous execution can be valuable for robot manipulation, enabling robots to operate more safely, determine when a behavior is inappropriate, and otherwise exhibit more common sense. By using multiple complementary sensory modalities, robots could potentially detect a wider variety of anomalies, such as anomalous contact or a loud utterance by a human. However, task variability and the potential for false positives make online anomaly detection challenging, especially for long-duration manipulation behaviors. In this paper, we provide evidence for the value of multimodal execution monitoring and the use of a detection threshold that varies based on the progress of execution. Using a data-driven approach, we train an execution monitor that runs in parallel to a manipulation behavior. Like previous methods for anomaly detection, our method trains a hidden Markov model (HMM) using multimodal observations from non-anomalous executions. In contrast to prior work, our system also uses a detection threshold that changes based on the execution progress. We evaluated our approach with haptic, visual, auditory, and kinematic sensing during a variety of manipulation tasks performed by a PR2 robot. The tasks included pushing doors closed, operating switches, and assisting ablebodied participants with eating yogurt. In our evaluations, our anomaly detection method performed substantially better with multimodal monitoring than single modality monitoring. It also resulted in more desirable ROC curves when compared with other detection threshold methods from the literature, obtaining higher true positive rates for comparable false positive rates.

A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-Based Variational Autoencoder

“…We introduce a varying threshold that changes over the estimated state of a task execution motivated by the dynamic threshold [3]. Depending on the state of task executions, reconstruction quality may vary.…”

“…Researchers often reduce the dimension or select features before applying a classifier. Our previous work used also selected 4 hand-engineered features from 3 modalities for a likelihood-based classifier, HMM-GP, using hidden Markov models (HMM) [3], [4]. However, the compressed or selected representations may be missing information relevant to anomaly detection.…”

Active robot-assisted feeding with a general-purpose mobile manipulator: Design, evaluation, and lessons learned

“…A workshop publication described an early, less-capable version of the meal-assistance system that required fiducial markers placed on the person's head and the bowl [28]. Otherwise, our publications involving meal-assistance have focused on execution monitoring [29,15,30,16]. The newer meal-assistance system that we present now was used in a conference paper [15] to evaluate an execution monitoring system, but the paper provided no details about the meal-assistance system.…”

Fast, robust, and versatile event detection through HMM belief state gradient measures

“…1) Pre-Recovery Performance: For pre-recovery performance computation, during each trial, we record the anomalies triggered by the testing metric and count its true positives, false positives and false negatives as illustrated in technique ID Accuracy AFF/DCC/CSM/SVM [26] 84.66% 1 sHDP-HMM [5] 89.50% RCBHT w/ multiclass SVM [18] 97.00% HMM w/GradientBased Measure [current] 98.40% Tool breakage SVM [13] 99.38% technique anomalyID Accuracy HMM,varying threshold [24] ∼ 80.00% MLP [8] 83.27% sHDP-VAR-HMM,mag metric [7] ∼ 85.00% sHDP-HMM [5] 87.50% RCBHT w/ multiclass SVM [18] 97.00% HMM, gradient metric (current) 100.00% technique reaction time sHDP-VAR-HMM,mag metric [7] 3.70% 2 HMM, gradient metric (current) 1.84%…”