Failure Detection Using Proprioceptive, Auditory and Visual Modalities

İnceoğlu, Arda; İnce, Gökhan; Yaslan, Yusuf; Sariel, Sanem

doi:10.1109/iros.2018.8594169

Cited by 18 publications

(22 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Experiments 4a,b,c, for anomaly identification, a total of 124 trials were used for testing. 11 For anomaly identification, we had an average accuracy of 97.04%, an average precision of 97.02% and an average recall of 99.42% across the three sub-experiments. Very strong performance was achieved all around and charted in Fig.…”

Section: Experiments 4: Adaptation Resultsmentioning

confidence: 99%

“…In [10], language is used to generate motion, and a simple HMM is used to detect success or failure based on trajectory position information. In [11], visual, audio, and proprioceptive features are used through Hidden Markov Models (HMMs) and other heuristics on a tabletop task to detect failure. In [12], a hierarchical Dirichlet process (HDP) prior was used with HMMs and a Gaussian observation model and Gibbs sampling to do anomaly iden-tification and multi-class classification.…”

Section: Anomaly Identification and Classificationmentioning

confidence: 99%

See 1 more Smart Citation

Endowing Robots with Longer-term Autonomy by Recovering from External Disturbances in Manipulation Through Grounded Anomaly Classification and Recovery Policies

Luo

Duan

et al. 2021

J Intell Robot Syst

View full text Add to dashboard Cite

Robots are poised to interact with humans in unstructured environments. Despite increasingly robust control algorithms, failure modes arise whenever the underlying dynamics are poorly modeled, especially in unstructured environments. We contribute a set of recovery policies to deal with anomalies produced by external disturbances. The recoveries work when various different types of anomalies are triggered any number of times at any point in the task, including during already running recoveries. Our recovery critic stands atop of a tightly-integrated, graph-based online motion-generation and introspection system. Policies, skills, and introspection models are learned incrementally and contextually over time. Recoveries are studied via a collaborative kitting task where a wide range of anomalous conditions are experienced in the system. We also contribute an extensive analysis of the performance of the tightly integrated anomaly identification, classification, and recovery system under extreme anomalous conditions. We show how the integration of such a system achieves performances greater than the sum of its parts.

show abstract

Section: Experiments 4: Adaptation Resultsmentioning

confidence: 99%

Section: Anomaly Identification and Classificationmentioning

confidence: 99%

Endowing Robots with Longer-term Autonomy by Recovering from External Disturbances in Manipulation Through Grounded Anomaly Classification and Recovery Policies

Luo

Duan

et al. 2021

J Intell Robot Syst

View full text Add to dashboard Cite

show abstract

“…They use force, sound and kinematic signals from a service robot to learn from nominal executions using hidden Markov models (HMMs) and Gaussian processes. Inceoglu et al [6] learn from several sensor modalities, using HMMs to classify extracted predicates from each modality into success and failure classes for different actions. They also present an end-to-end convolutional neural network [11], which classifies executions as success or failure, and identifies the failure types as well.…”

Section: A Execution Monitoring In Roboticsmentioning

confidence: 99%

“…Depending on the nature of the failure, different sensors might be used for detection; for example, force-torque sensors for collisions, or vision and auditory sensors for external events such as objects falling. Some methods [4], [5], [6] only make use of visual data such as RGB frames, depth frames, or the output of object detection algorithms. In this paper, we propose a method for visual execution monitoring, using the videos from the robot's camera and the robot's kinematics.…”

Section: Introductionmentioning

confidence: 99%

Using Visual Anomaly Detection for Task Execution Monitoring

Thoduka,

Gall,

Plöger

2021

Preprint

View full text Add to dashboard Cite

Execution monitoring is essential for robots to detect and respond to failures. Since it is impossible to enumerate all failures for a given task, we learn from successful executions of the task to detect visual anomalies during runtime. Our method learns to predict the motions that occur during the nominal execution of a task, including camera and robot body motion. A probabilistic U-Net architecture is used to learn to predict optical flow, and the robot's kinematics and 3D model are used to model camera and body motion. The errors between the observed and predicted motion are used to calculate an anomaly score. We evaluate our method on a dataset of a robot placing a book on a shelf, which includes anomalies such as falling books, camera occlusions, and robot disturbances. We find that modeling camera and body motion, in addition to the learning-based optical flow prediction, results in an improvement of the area under the receiver operating characteristic curve from 0.752 to 0.804, and the area under the precision-recall curve from 0.467 to 0.549.

show abstract

“…[18] also introduces the tactile information to improve the auditory classification performance. In [19], the sound information can work together with visual information to detect failure in robotic manipulation. Additionally, a type of audiovisual embodied navigation task is proposed recently, in which the agent navigates to a sounding object by leveraging both visual and auditory data [20][21].…”

Section: Introductionmentioning

confidence: 99%

Audio-Visual Grounding Referring Expression for Robotic Manipulation

Wang¹,

Wang²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Referring expressions are commonly used when referring to a specific target in people's daily dialogue. In this paper, we develop a novel task of audio-visual grounding referring expression for robotic manipulation. The robot leverages both the audio and visual information to understand the referring expression in the given manipulation instruction and the corresponding manipulations are implemented. To solve the proposed task, an audio-visual framework is proposed for visual localization and sound recognition. We have also established a dataset which contains visual data, auditory data and manipulation instructions for evaluation. Finally, extensive experiments are conducted both offline and online to verify the effectiveness of the proposed audio-visual framework. And it is demonstrated that the robot performs better with the audiovisual data than with only the visual data.

show abstract

Failure Detection Using Proprioceptive, Auditory and Visual Modalities

Cited by 18 publications

References 23 publications

Endowing Robots with Longer-term Autonomy by Recovering from External Disturbances in Manipulation Through Grounded Anomaly Classification and Recovery Policies

Endowing Robots with Longer-term Autonomy by Recovering from External Disturbances in Manipulation Through Grounded Anomaly Classification and Recovery Policies

Using Visual Anomaly Detection for Task Execution Monitoring

Audio-Visual Grounding Referring Expression for Robotic Manipulation

Contact Info

Product

Resources

About