Using Visual Anomaly Detection for Task Execution Monitoring

Thoduka, Santosh; Gall, Jüergen; Plöger, Paul G.

doi:10.1109/iros51168.2021.9636133

Cited by 4 publications

(2 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An advantage of using RGB video lies in the availability of many well-established algorithms that can be used to extract auxiliary information to aid the anomaly detection. An example is the frequent use of optical flow to help detecting motion-based anomalies [11,13,22,33]. Some works use much higher-level information, e.g.…”

Section: Related Workmentioning

confidence: 99%

Unsupervised Anomaly Detection from Time-of-Flight Depth Images

Schneider¹,

Rambach²,

Mirbach³

et al. 2022

Preprint

View full text Add to dashboard Cite

Video anomaly detection (VAD) addresses the problem of automatically finding anomalous events in video data. The primary data modalities on which current VAD systems work on are monochrome or RGB images. Using depth data in this context instead is still hardly explored in spite of depth images being a popular choice in many other computer vision research areas and the increasing availability of inexpensive depth camera hardware. We evaluate the application of existing autoencoder-based methods on depth video and propose how the advantages of using depth data can be leveraged by integration into the loss function. Training is done unsupervised using normal sequences without need for any additional annotations. We show that depth allows easy extraction of auxiliary information for scene analysis in the form of a foreground mask and demonstrate its beneficial effect on the anomaly detection performance through evaluation on a large public dataset, for which we are also the first ones to present results on.

show abstract

Section: Related Workmentioning

confidence: 99%

Unsupervised Anomaly Detection from Time-of-Flight Depth Images

Schneider¹,

Rambach²,

Mirbach³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The authors adopt LSTM-based variational autoencoders to process multimodal input from a sensor set including a camera, a microphone, a joint encoder, and a force sensor. In another work [36], multimodal cues are used to detect book manipulation failures on shelves. [37] fuses visuo-tactile cues for grasp failure detection.…”

Section: Related Workmentioning

confidence: 99%

FINO-Net: A Deep Multimodal Sensor Fusion Framework for Manipulation Failure Detection

İnceoğlu

Aksoy

et al. 2021

2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

An autonomous service robot should be able to interact with its environment safely and robustly without requiring human assistance. Unstructured environments are challenging for robots since the exact prediction of outcomes is not always possible. Even when the robot behaviors are well-designed, the unpredictable nature of physical robotobject interaction may prevent success in object manipulation. Therefore, execution of a manipulation action may result in an undesirable outcome involving accidents or damages to the objects or environment. Situation awareness becomes important in such cases to enable the robot to (i) maintain the integrity of both itself and the environment, (ii) recover from failed tasks in the short term, and (iii) learn to avoid failures in the long term. For this purpose, robot executions should be continuously monitored, and failures should be detected and classified appropriately. In this work, we focus on detecting and classifying both manipulation and post-manipulation phase failures using the same exteroception setup. We cover a diverse set of failure types for primary tabletop manipulation actions. In order to detect these failures, we propose FINO-Net [1], a deep multimodal sensor fusion based classifier network. Proposed network accurately detects and classifies failures from raw sensory data without any prior knowledge. In this work, we use our extended FAILURE dataset [1] with 99 new multimodal manipulation recordings and annotate them with their corresponding failure types. FINO-Net achieves 0.87 failure detection and 0.80 failure classification F1 scores. Experimental results show that proposed architecture is also appropriate for real-time use.

show abstract