Introspective Evaluation of Perception Performance for Parameter Tuning without Ground Truth

Hu, Humphrey; Kantor, George

doi:10.15607/rss.2017.xiii.033

Cited by 9 publications

(5 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The concept of introspection in robotics is first introduced in [ 16 ] as a self-assessment mechanism for a robot to assist its decision-making. In later years, this concept has been interpreted and applied for failure prediction in different applications [ 16 , 17 , 21 , 22 , 23 ]. For example, Daftry et al [ 17 ] described the robot introspection as the self-evaluating facility for a robot system to know when it does not know.…”

Section: Related Workmentioning

confidence: 99%

Introspective False Negative Prediction for Black-Box Object Detectors in Autonomous Driving

Yang

Chen

et al. 2021

Sensors

View full text Add to dashboard Cite

Object detection plays a critical role in autonomous driving, but current state-of-the-art object detectors will inevitably fail in many driving scenes, which is unacceptable for safety-critical automated vehicles. Given the complexity of the real traffic scenarios, it is impractical to guarantee zero detection failure; thus, online failure prediction is of crucial importance to mitigate the risk of traffic accidents. Of all the failure cases, False Negative (FN) objects are most likely to cause catastrophic consequences, but little attention has been paid to the online FN prediction. In this paper, we propose a general introspection framework that can make online prediction of FN objects for black-box object detectors. In contrast to existing methods which rely on empirical assumptions or handcrafted features, we facilitate the FN feature extraction by an introspective FN predictor we designed in this framework. For this purpose, we extend the original concept of introspection to object-wise FN predictions, and propose a multi-branch cooperation mechanism to address the distinct foreground-background imbalance problem of FN objects. The effectiveness of the proposed framework is verified through extensive experiments and analysis, and the results show that our method successfully predicts the FN objects with 81.95% precision for 88.10% recall on the challenging KITTI Benchmark, and effectively improves object detection performance by taking FN predictions into consideration.

show abstract

Section: Related Workmentioning

confidence: 99%

Introspective False Negative Prediction for Black-Box Object Detectors in Autonomous Driving

Yang

Chen

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…They also anticipate when to hand over control to the human user. [15] proposed a method to evaluate the performance of a perception system without any ground truth. [16] argued that most vision based perception failure occurs because of improper illumination of the scene.…”

Section: Related Workmentioning

confidence: 99%

Did You Miss the Sign? A False Negative Alarm System for Traffic Sign Detectors

Rahman

Sünderhauf²,

Dayoub³

2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Object detection is an integral part of an autonomous vehicle for its safety-critical and navigational purposes. Traffic signs as objects play a vital role in guiding such systems. However, if the vehicle fails to locate any critical sign, it might make a catastrophic failure. In this paper, we are proposing an approach to identify traffic signs that have been mistakenly discarded by the object detector. The proposed method raises an alarm when it discovers a failure by the object detector to detect a traffic sign. This approach can be useful to evaluate the performance of the detector during the deployment phase. We trained a single shot multi-box object detector to detect traffic signs and used its internal features to train a separate false negative detector (FND). During deployment, FND decides whether the traffic sign detector (TSD) has missed a sign or not. We are using precision and recall to measure the accuracy of FND in two different datasets. For 80% recall, FND has achieved 89.9% precision in Belgium Traffic Sign Detection dataset and 90.8% precision in German Traffic Sign Recognition Benchmark dataset respectively. To the best of our knowledge, our method is the first to tackle this critical aspect of false negative detection in robotic vision. Such a fail-safe mechanism for object detection can improve the engagement of robotic vision systems in our daily life.

show abstract

“…They evaluate the inherent uncertainty measure of these models by inspecting their changes when the model is exposed to new unseen data. Hu et al [4] tune the parameters of a localization algorithm by means of minimizing the inherent uncertainty measure of their perception model. Using only the underlying uncertainty of the perception model limits the introspective capacity of the system.…”

Section: Related Workmentioning

confidence: 99%

IVOA: Introspective Vision for Obstacle Avoidance

Rabiee

Biswas

2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Vision, as an inexpensive yet information rich sensor, is commonly used for perception on autonomous mobile robots. Unfortunately, accurate vision-based perception requires a number of assumptions about the environment to hold -some examples of such assumptions, depending on the perception algorithm at hand, include purely lambertian surfaces, texture-rich scenes, absence of aliasing features, and refractive surfaces. In this paper, we present an approach for introspective vision for obstacle avoidance (IVOA) -by leveraging a supervisory sensor that is occasionally available, we detect failures of stereo vision-based perception from divergence in plans generated by vision and the supervisory sensor. By projecting the 3D coordinates where the plans agree and disagree onto the images used for vision-based perception, IVOA generates a training set of reliable and unreliable image patches for perception. We then use this training dataset to learn a model of which image patches are likely to cause failures of the vision-based perception algorithm. Using this model, IVOA is then able to predict whether the relevant image patches in the observed images are likely to cause failures due to vision (both false positives and false negatives). We empirically demonstrate with extensive real-world data from both indoor and outdoor environments, the ability of IVOA to accurately predict the failures of two distinct vision algorithms.

show abstract

Introspective Evaluation of Perception Performance for Parameter Tuning without Ground Truth

Cited by 9 publications

References 28 publications

Introspective False Negative Prediction for Black-Box Object Detectors in Autonomous Driving

Introspective False Negative Prediction for Black-Box Object Detectors in Autonomous Driving

Did You Miss the Sign? A False Negative Alarm System for Traffic Sign Detectors

IVOA: Introspective Vision for Obstacle Avoidance

Contact Info

Product

Resources

About