Self-Supervised Domain Mismatch Estimation for Autonomous Perception

Löhdefink, Jonas; Fehrling, Justin; Klingner, Marvin; Hüger, Fabian; Schlicht, Peter; Schmidt, Nico M.; Fingscheidt, Tim

doi:10.1109/cvprw50498.2020.00175

Cited by 19 publications

(11 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Among online-capable metrics for the overall performance prediction, some only focus at malfunction detection and correction (again involving an ensemble of DNNs) [17], [64], or exploit temporal inconsistency between consecutive predictions [15], which has to be defined in a highly task-specific way. The closest prior work to ours is presumably from Löhdefink et al [18], who propose to train an autoencoder to reconstruct an image on the same data a semantic segmentation DNN is trained on, showing a correlation between both task's metrics. Note, however, that the method from [18] is only applicable when using statistics over a large number of images.…”

Section: Performance Prediction Of Neural Networkmentioning

confidence: 99%

“…The closest prior work to ours is presumably from Löhdefink et al [18], who propose to train an autoencoder to reconstruct an image on the same data a semantic segmentation DNN is trained on, showing a correlation between both task's metrics. Note, however, that the method from [18] is only applicable when using statistics over a large number of images. While the mentioned works propose additional metrics for the online performance evaluation of DNNs, none of them shows the capability to operate on a single-image basis to predict and quantify a DNN's absolute performance, which our method indeed is capable of with good fidelity.…”

Section: Performance Prediction Of Neural Networkmentioning

confidence: 99%

“…one typically assumes that an offline-measured performance of a DNN is also valid in inference, this is actually not true due to the mentioned environment changes. Meanwhile, less-frequently proposed online-capable algorithms are either task-specific [15], rely on ensembles of DNNs [16], [17], or only show the correlation of a proposed metric to the absolute performance metric without further outlining an online-capable predictive scheme [15], [18]. Naively using the confidence scores of the network itself [19] is not recommended as DNNs often assign a probability of close to one to a single class [20], and even more important, the uncertainty of measurements does not bear predictive power to estimate the absolute DNN performance.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Online Performance Prediction of Perception DNNs by Multi-Task Learning With Depth Estimation

Klingner

Fingscheidt

2021

IEEE Trans. Intell. Transport. Syst.

Self Cite

View full text Add to dashboard Cite

Online performance prediction (or: observation) of deep neural networks (DNNs) in highly automated driving presents an unsolved task until now, as most DNNs are evaluated offline requiring datasets with ground truth labels. In practice, however, DNN performance depends on the used camera type, lighting and weather conditions, and on various other kinds of domain shift. Also, the input to DNN-based perception systems can be perturbed by adversarial attacks requiring means to detect these input perturbations. In this work we propose a method to mitigate these problems by a multi-task learning approach with monocular depth estimation as a secondary task, which enables us to predict the DNN's performance for various other (primary) tasks by evaluating only the depth estimation task with a physical depth measurement provided, e.g., by a LiDAR sensor. We show the effectiveness of our method for the primary task of semantic segmentation using various training datasets, test datasets, model architectures, and input perturbations. Our method provides an effective way to predict (observe) the performance of DNNs for semantic segmentation even on a single-image basis and is transferable to other primary DNN-based perception tasks in a straightforward manner.

show abstract

Section: Performance Prediction Of Neural Networkmentioning

confidence: 99%

Section: Performance Prediction Of Neural Networkmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Online Performance Prediction of Perception DNNs by Multi-Task Learning With Depth Estimation

Klingner

Fingscheidt

2021

IEEE Trans. Intell. Transport. Syst.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Alternatively, temporal consistency of consecutive predictions can be checked [55], though the definition of consistency is highly task-specific. A recent successful approach of Löhdefink et al [34] estimates the performance of a semantic segmentation model based on the image reconstruction error of an autoencoder DNN. Similarly, Klingner et al [30] use the depth estimation task for this purpose.…”

Section: Related Workmentioning

confidence: 99%

“…Some initial works intend to mitigate the problem by temporal or network ensemble consistency checks [17,51,55], however, these approaches are not straightforward extendable to arbitrary tasks. Another approach uses an autoencoder to predict the performance based on the image reconstruction error [34]. This has, however, not been shown to be applicable on a few-shot basis, which would be necessary for online operation.…”

Section: Introductionmentioning

confidence: 99%

Improving Online Performance Prediction for Semantic Segmentation

Klingner¹,

Bär²,

Mross³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

In this work we address the task of observing the performance of a semantic segmentation deep neural network (DNN) during online operation, i.e., during inference, which is of high importance in safety-critical applications such as autonomous driving. Here, many high-level decisions rely on such DNNs, which are usually evaluated offline, while their performance in online operation remains unknown. To solve this problem, we propose an improved online performance prediction scheme, building on a recently proposed concept of predicting the primary semantic segmentation task's performance. This can be achieved by evaluating the auxiliary task of monocular depth estimation with a measurement supplied by a LiDAR sensor and a subsequent regression to the semantic segmentation performance. In particular, we propose (i) sequential training methods for both tasks in a multi-task training setup, (ii) to share the encoder as well as parts of the decoder between both task's networks for improved efficiency, and (iii) a temporal statistics aggregation method, which significantly reduces the performance prediction error at the cost of a small algorithmic latency. Evaluation on the KITTI dataset shows that all three aspects improve the performance prediction compared to previous approaches.

show abstract

Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety

Houben

Abrecht

Akila

et al. 2022

Deep Neural Networks and Data for Automated Driving

Self Cite

View full text Add to dashboard Cite

Deployment of modern data-driven machine learning methods, most often realized by deep neural networks (DNNs), in safety-critical applications such as health care, industrial plant control, or autonomous driving is highly challenging due to numerous model-inherent shortcomings. These shortcomings are diverse and range from a lack of generalization over insufficient interpretability and implausible predictions to directed attacks by means of malicious inputs. Cyber-physical systems employing DNNs are therefore likely to suffer from so-called safety concerns, properties that preclude their deployment as no argument or experimental setup can help to assess the remaining risk. In recent years, an abundance of state-of-the-art techniques aiming to address these safety concerns has emerged. This chapter provides a structured and broad overview of them. We first identify categories of insufficiencies to then describe research activities aiming at their detection, quantification, or mitigation. Our work addresses machine learning experts and safety engineers alike: The former ones might profit from the broad range of machine learning topics covered and discussions on limitations of recent methods. The latter ones might gain insights into the specifics of modern machine learning methods. We hope that this contribution fuels discussions on desiderata for machine learning systems and strategies on how to help to advance existing approaches accordingly.

show abstract

Self-Supervised Domain Mismatch Estimation for Autonomous Perception

Cited by 19 publications

References 40 publications

Online Performance Prediction of Perception DNNs by Multi-Task Learning With Depth Estimation

Online Performance Prediction of Perception DNNs by Multi-Task Learning With Depth Estimation

Improving Online Performance Prediction for Semantic Segmentation

Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety

Contact Info

Product

Resources

About