Introspective perception: Learning to predict failures in vision systems

Daftry, Shreyansh; Zeng, Sam; Bagnell, J. Andrew; Hebert, Martial

doi:10.1109/iros.2016.7759279

Cited by 65 publications

(41 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To tackle this problem, they proposed a failure detection and recovery maneuver for a vision system. A system agnostic framework has been proposed by [17] to predict failure in a vision system. They argued that predicting failure from raw sensor data is more effective than using the uncertainty of model-based classifiers.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Did You Miss the Sign? A False Negative Alarm System for Traffic Sign Detectors

Rahman

Sünderhauf²,

Dayoub³

2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Object detection is an integral part of an autonomous vehicle for its safety-critical and navigational purposes. Traffic signs as objects play a vital role in guiding such systems. However, if the vehicle fails to locate any critical sign, it might make a catastrophic failure. In this paper, we are proposing an approach to identify traffic signs that have been mistakenly discarded by the object detector. The proposed method raises an alarm when it discovers a failure by the object detector to detect a traffic sign. This approach can be useful to evaluate the performance of the detector during the deployment phase. We trained a single shot multi-box object detector to detect traffic signs and used its internal features to train a separate false negative detector (FND). During deployment, FND decides whether the traffic sign detector (TSD) has missed a sign or not. We are using precision and recall to measure the accuracy of FND in two different datasets. For 80% recall, FND has achieved 89.9% precision in Belgium Traffic Sign Detection dataset and 90.8% precision in German Traffic Sign Recognition Benchmark dataset respectively. To the best of our knowledge, our method is the first to tackle this critical aspect of false negative detection in robotic vision. Such a fail-safe mechanism for object detection can improve the engagement of robotic vision systems in our daily life.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Baseline 2: Similar approaches like [12], [17] is adopted for the baseline 2 training. TSD is used to detect traffic signs from the BTSD training split, and we collect proposals where TSD score is less than λ.…”

Section: Baselinesmentioning

confidence: 99%

Did You Miss the Sign? A False Negative Alarm System for Traffic Sign Detectors

Rahman

Sünderhauf²,

Dayoub³

2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

show abstract

“…They test their method for two different tasks of image classification and image segmentation. Daftry et al [2] train a convolutional neural network (CNN) that uses both still images and optical flow frames to predict the probability of failure for the navigation system of an actual UAV. A follow up work [5] trains an SVM classifier to choose the best recovery action when the vision system has high uncertainty.…”

Section: Related Workmentioning

confidence: 99%

“…Second, the extracted information can be used effectively to discover and categorize sources of errors for the vision system. While previous works on introspective vision systems [1], [2] output a single failure probability score for the whole input image, IVOA, to the best of our knowledge, is the first to digest the input image in detail and localize the potential sources of error and their type.…”

Section: Introductionmentioning

confidence: 99%

IVOA: Introspective Vision for Obstacle Avoidance

Rabiee

Biswas

2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Vision, as an inexpensive yet information rich sensor, is commonly used for perception on autonomous mobile robots. Unfortunately, accurate vision-based perception requires a number of assumptions about the environment to hold -some examples of such assumptions, depending on the perception algorithm at hand, include purely lambertian surfaces, texture-rich scenes, absence of aliasing features, and refractive surfaces. In this paper, we present an approach for introspective vision for obstacle avoidance (IVOA) -by leveraging a supervisory sensor that is occasionally available, we detect failures of stereo vision-based perception from divergence in plans generated by vision and the supervisory sensor. By projecting the 3D coordinates where the plans agree and disagree onto the images used for vision-based perception, IVOA generates a training set of reliable and unreliable image patches for perception. We then use this training dataset to learn a model of which image patches are likely to cause failures of the vision-based perception algorithm. Using this model, IVOA is then able to predict whether the relevant image patches in the observed images are likely to cause failures due to vision (both false positives and false negatives). We empirically demonstrate with extensive real-world data from both indoor and outdoor environments, the ability of IVOA to accurately predict the failures of two distinct vision algorithms.

show abstract

“…Much of this work has been done on vision systems, for instance predicting segmentation or horizon detection failures [25], or traversability estimation failures [26]. Other work has shown prediction of the heuristic performance of a vision-based navigation system [17], [18], and of a classification system [27].…”

Section: Introspectionmentioning

confidence: 99%

Introspective Evaluation of Perception Performance for Parameter Tuning without Ground Truth

Hu¹,

Kantor²

2017

Robotics: Science and Systems XIII

View full text Add to dashboard Cite

Abstract-Modern perception systems are notoriously complex, featuring dozens of interacting parameters that must be tuned to achieve good performance. Conventional tuning approaches require expensive ground truth, while heuristic methods are difficult to generalize. In this work, we propose an introspective ground-truth-free approach to evaluating the performance of a generic perception system. By using the posterior distribution estimate generated by a Bayesian estimator, we show that the expected performance can be estimated efficiently and without ground truth. Our simulated and physical experiments in a demonstrative indoor ground robot state estimation application show that our approach can order parameters similarly to using a ground-truth system, and is able to accurately identify top-performing parameters in varying contexts. In contrast, baseline approaches that reason only about observation log-likelihood fail in the face of challenging perceptual phenomena.

show abstract

Introspective perception: Learning to predict failures in vision systems

Cited by 65 publications

References 40 publications

Did You Miss the Sign? A False Negative Alarm System for Traffic Sign Detectors

Did You Miss the Sign? A False Negative Alarm System for Traffic Sign Detectors

IVOA: Introspective Vision for Obstacle Avoidance

Introspective Evaluation of Perception Performance for Parameter Tuning without Ground Truth

Contact Info

Product

Resources

About