Introspective classification for robot perception

Abstract

In robotics, the use of a classification framework which produces scores with inappropriate confidences will ultimately lead to the robot making dangerous decisions. In order to select a framework which will make the best decisions, we should pay careful attention to the ways in which it generates scores. Precision and recall have been widely adopted as canonical metrics to quantify the performance of learning algorithms, but for robotics applications involving mission-critical decision making, good performance in relation to these metrics is insufficient. We introduce and motivate the importance of a classifier’s introspective capacity: the ability to associate an appropriate assessment of confidence with any test case. We propose that a key ingredient for introspection is a framework’s potential to increase its uncertainty with the distance between a test datum its training data. We compare the introspective capacities of a number of commonly used classification frameworks in both classification and detection tasks, and show that better introspection leads to improved decision making in the context of tasks such as autonomous driving or semantic map generation.

Safe Visual Navigation via Deep Learning and Novelty Detection

“…Grimmett et al [12] suggest that if a classification error is to be made in robotic decision making, it should be made with high reported uncertainty so that the robot can avoid consequences of a wrong decision. Sofman et al [32] use online novelty detection in this way, to avoid scenarios for which their robot is untrained.…”

IVOA: Introspective Vision for Obstacle Avoidance

“…One line of work tries to use the inherent uncertainty measure provided by the vision model. Grimmet et al [3] look into probabilistic function approximators such as Gaussian processes as well as boot strapped classifiers that use the consensus of an ensemble of models as a measure of confidence. They evaluate the inherent uncertainty measure of these models by inspecting their changes when the model is exposed to new unseen data.…”

Introspective Evaluation of Perception Performance for Parameter Tuning without Ground Truth

“…Bayesian inference techniques such as the ubiquitous Kalman filter provide an estimate of this distribution, which we will denote asp(x|ξ). It may be possible to adapt non-Bayesian techniques, for instance SVMs for classification, by introducing introspection as in [23].…”

“…One work reasons about classification output uncertainty by considering distributions of models [23]. This is similar to the older concept of filtering optimism, a condition wherein a filter becomes overly confident in its estimate [24].…”

“…Our work is similar to [23] in that it uses distributions of hypotheses to introspect performance without ground truth. However, our approach operates on the posterior distribution over latents and observations directly, as opposed to over model hypotheses.…”