Simultaneous Semantic Segmentation and Outlier Detection in Presence of Domain Shift

Abstract

Recent success on realistic road driving datasets has increased interest in exploring robust performance in real-world applications. One of the major unsolved problems is to identify image content which can not be reliably recognized with a given inference engine. We therefore study approaches to recover a dense outlier map alongside the primary task with a single forward pass, by relying on shared convolutional features. We consider semantic segmentation as the primary task and perform extensive validation on WildDash val (inliers), LSUN val (outliers), and pasted objects from Pascal VOC 2007 (outliers). We achieve the best validation performance by training to discriminate inliers from pasted ImageNet-1k content, even though ImageNet-1k contains many roaddriving pixels, and, at least nominally, fails to account for the full diversity of the visual world. The proposed two-head model performs comparably to the C-way multi-class model trained to predict uniform distribution in outliers, while outperforming several other validated approaches. We evaluate our best two models on the WildDash test dataset and set a new state of the art on the WildDash benchmark.

ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding

“…Existing works that model uncertainty in semantic segmentation [1,21] are evaluated only with IoU, which does not assess the predicted confidence. In contrast, for uncertainty-aware semantic segmentation, algorithms are required to output both a hard semantic prediction Ĥ and a confidence map C with values in the range [0, 1].…”

“…We have also trained [1] on ACDC, using the GT invalid masks for training its outlier detection part. The learned confidence by [1] leads to lower test set AUIoU (52.0%) than constant confidence (53.0%), indicating that a better modeling of uncertainty is needed in future approaches.…”

Map-Guided Curriculum Domain Adaptation and Uncertainty-Aware Evaluation for Semantic Nighttime Image Segmentation

“…In contrast, we introduce UIoU, a new semantic segmentation metric that handles images with regions of uncertain semantic content and is suited for adverse conditions. Our uncertainty-aware evaluation is complementary to uncertaintyaware methods such as [56] and [57] that explicitly incorporate uncertainty in their model formulation and aims to promote the development of such methods, as UIoU rewards models that accurately capture heteroscedastic aleatoric uncertainty [56] in the input images through the different treatment of invalid and valid regions.…”

“…This implies that on Dark Zurich-test, these models generally have lower confidence on invalid regions than valid ones, although they do not explicitly model uncertainty, as there is no distinction between valid and invalid regions during training. Modeling confidence explicitly along the lines of [56], [57] could further increase UIoU and is an interesting direction for future work. Second, the comparative performance of the methods is the same across all values of θ, except the pair of MGCDA and GCMA.…”

One Versus All for Deep Neural Network for Uncertainty (OVNNI) Quantification

“…Zendel et al [64] propose a semantic segmentation dataset for checking the confidence score of DNNs. The authors of [3] train a DNN to predict OOD confidence score. Lee et al [36] train a GAN along with the classifier to produce near-distribution examples and enforce lower classifier confidence on GAN samples.…”