Uncertainty Measures and Prediction Quality Rating for the Semantic Segmentation of Nested Multi Resolution Street Scene Images

Abstract

In the semantic segmentation of street scenes the reliability of the prediction and therefore uncertainty measures are of highest interest. We present a method that generates for each input image a hierarchy of nested crops around the image center and presents these, all re-scaled to the same size, to a neural network for semantic segmentation. The resulting softmax outputs are then post processed such that we can investigate mean and variance over all image crops as well as mean and variance of uncertainty heat maps obtained from pixel-wise uncertainty measures, like the entropy, applied to each crop's softmax output. In our tests, we use the publicly available DeepLabv3+ MobilenetV2 network (trained on the Cityscapes dataset) and demonstrate that the incorporation of crops improves the quality of the prediction and that we obtain more reliable uncertainty measures. These are then aggregated over predicted segments for either classifying between IoU = 0 and IoU > 0 (meta classification) or predicting the IoU via linear regression (meta regression). The latter yields reliable performance estimates for segmentation networks, in particular useful in the absence of ground truth. For the task of meta classification we obtain a classification accuracy of 81.93% and an AUROC of 89.89%. For meta regression we obtain an R 2 value of 84.77%. These results yield significant improvements compared to other approaches.

Gradient-Based Quantification of Epistemic Uncertainty for Deep Object Detectors

“…Since then, the approach has been applied to semantic segmentation e.g. in [31,32,33,34,35], instance segmentation in videos [36] and object detection [19,37]. The common idea in all of these adaptations is the usage of lightweight classification and regression models to explicitly learn the map between uncertainty and the labels TP/FP or between uncertainty and IoU .…”

Spatially Varying Label Smoothing: Capturing Uncertainty from Expert Annotations

“…Center point replaced by the sum of neighbors and normalized dense pixel-wise classification, it is important to consider that the individual predictions are actually not independent, and neither are the class labels. The likelihood of the co-occurrence of different classes is spatially varying, in particular, near object boundaries which typically exhibit higher uncertainty [6,28,31,34].…”

Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation

“…They used a probabilistic U-Net to quantify uncertainty in the segmentation of lung abnormalities. Rottmann and Schubert [30] proposed a prediction quality rat-ing method for segmentation of nested multi-resolution street scene images by measuring both pixel-wise and segment-wise measures of uncertainty as predictive metrics for segmentation quality. Recently, Karimi et al [31] used ensembling for uncertainty estimation of difficult to segment regions and used this information to improve clinical target volume estimation in prostate ultrasound images.…”

“…The metric can be used to detect out-of-distribution samples and hard or ambiguous cases. Such metrics have been previously proposed for street scene segmentation [30]. Given the pixel-level class predictions ŷi and their associated ground truth class y i for a predicted segment Ŝk = {s ∈ (x i , ŷi )|ŷ i = k}, we propose to use the average of pixel-wise entropy values over the predicted foreground segment Ŝk as a scalar metric for volume-level confidence of that segment as:…”