Online Performance Prediction of Perception DNNs by Multi-Task Learning With Depth Estimation

Abstract

Online performance prediction (or: observation) of deep neural networks (DNNs) in highly automated driving presents an unsolved task until now, as most DNNs are evaluated offline requiring datasets with ground truth labels. In practice, however, DNN performance depends on the used camera type, lighting and weather conditions, and on various other kinds of domain shift. Also, the input to DNN-based perception systems can be perturbed by adversarial attacks requiring means to detect these input perturbations. In this work we propose a method to mitigate these problems by a multi-task learning approach with monocular depth estimation as a secondary task, which enables us to predict the DNN's performance for various other (primary) tasks by evaluating only the depth estimation task with a physical depth measurement provided, e.g., by a LiDAR sensor. We show the effectiveness of our method for the primary task of semantic segmentation using various training datasets, test datasets, model architectures, and input perturbations. Our method provides an effective way to predict (observe) the performance of DNNs for semantic segmentation even on a single-image basis and is transferable to other primary DNN-based perception tasks in a straightforward manner.

Improving Online Performance Prediction for Semantic Segmentation

“…A recent successful approach of Löhdefink et al [34] estimates the performance of a semantic segmentation model based on the image reconstruction error of an autoencoder DNN. Similarly, Klingner et al [30] use the depth estimation task for this purpose. Both approaches, however, still have a high error in their segmentation performance predictions, which we significantly reduce in this work.…”

“…Both approaches, however, still have a high error in their segmentation performance predictions, which we significantly reduce in this work. In contrast to [34,30] architecture is best suited for performance prediction. Also, we show how and under which conditions it is useful to employ a temporal aggregation of performance predictions.…”

“…The loss function is then based on the difference between the projected images x t →t and the actual image x t . As previous approaches [20,30], we use a mixture of absolute difference and structural similarity (SSIM) difference weighted by a factor α = 0.85. The absolute difference • 1 is computed over all color channels, while the SSIM difference SSIM i (•) ∈ I is additionally computed on 3 × 3 patches around the pixel index i of the images:…”

“…Additionally, we follow recent works in imposing a smoothness loss [19], allowing sharp depth boundaries only at pixel positions with strong color changes inside the input image. This additional loss is weighted by a factor of 10 −3 as in [5,19,30]. Also, we apply the automasking technique from [20] to minimize the influence of non-moving dynamic objects or frames.…”

“…Addressing this task of online performance prediction, we build on the work of Klingner et al [30], proposing a method, which relies on the auxiliary task of depth estimation also predicted by a DNN. As shown in Fig.…”