FINO-Net: A Deep Multimodal Sensor Fusion Framework for Manipulation Failure Detection

İnceoğlu, Arda; Aksoy, Eren Erdal; Ak, Abdullah Cihan; Sariel, Sanem

doi:10.48550/arxiv.2011.05817

Cited by 1 publication

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Inceoglu et al [6] learn from several sensor modalities, using HMMs to classify extracted predicates from each modality into success and failure classes for different actions. They also present an end-to-end convolutional neural network [11], which classifies executions as success or failure, and identifies the failure types as well. Wang et al [10] present a visual-tactile grasp dataset, consisting of tactile, joint and visual data of a robot arm grasping several objects.…”

Section: A Execution Monitoring In Roboticsmentioning

confidence: 99%

“…Since deep learning dominates most recent approaches for video anomaly detection, large-scale datasets are needed and have been published in recent years [8], [9]. Datasets for robotics also exist [10], [11], and are usually recorded with a single robot, and are task-specific. Generalizing learning models across robots requires large-scale datasets, which can be quite expensive [12].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Using Visual Anomaly Detection for Task Execution Monitoring

Thoduka,

Gall,

Plöger

2021

Preprint

View full text Add to dashboard Cite

Execution monitoring is essential for robots to detect and respond to failures. Since it is impossible to enumerate all failures for a given task, we learn from successful executions of the task to detect visual anomalies during runtime. Our method learns to predict the motions that occur during the nominal execution of a task, including camera and robot body motion. A probabilistic U-Net architecture is used to learn to predict optical flow, and the robot's kinematics and 3D model are used to model camera and body motion. The errors between the observed and predicted motion are used to calculate an anomaly score. We evaluate our method on a dataset of a robot placing a book on a shelf, which includes anomalies such as falling books, camera occlusions, and robot disturbances. We find that modeling camera and body motion, in addition to the learning-based optical flow prediction, results in an improvement of the area under the receiver operating characteristic curve from 0.752 to 0.804, and the area under the precision-recall curve from 0.467 to 0.549.

show abstract

Section: A Execution Monitoring In Roboticsmentioning

confidence: 99%