Improved Noise and Attack Robustness for Semantic Segmentation by Using Multi-Task Training with Self-Supervised Depth Estimation

Klingner, Marvin; Bär, Andreas; Fingscheidt, Tim

doi:10.1109/cvprw50498.2020.00168

Cited by 45 publications

(36 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While apparently most works in MTL of semantic segmentation and depth estimation focus on performance gains, Klingner et al [55] showed that the joint training process also has a positive effect on a segmentation model's robustness. In this work, we also do not focus on performance gains through MTL with depth estimation, but rather show that the joint learning process has a positive effect on the correlation between the different learned task's performance metrics.…”

Section: B Multi-task Learningmentioning

confidence: 99%

“…Perception tasks relying on ground truth labels during training are usually evaluated offline [13], [14], [56] on a labeled test dataset, which is not possible in an online setting. While many approaches aim at more robust training methods for DNNs [55], [57], such that the DNN is, e.g., more robust against adversarial attacks [9], [10], [58] or changes in the camera setup/weather conditions [59], [60], up to now there is no possibility to prevent severe performance degradation in arbitrary situations, facilitating research in online performance prediction (observation) methods.…”

Section: Performance Prediction Of Neural Networkmentioning

confidence: 99%

“…To sum up, we use the multi-task network from Klingner et al [55] (cf. Section II-B) to obtain estimates for both depth and segmentation, where the semantic segmentation is trained in a supervised fashion and the depth estimation in a self-supervised fashion as proposed by Godard et al [4] (cf.…”

Section: Original Contributionmentioning

confidence: 99%

“…Section II-A). To implement our novel performance prediction scheme we evaluate the depth estimation using a commonly used depth metric [4], [23], [37], [55] and predict the segmentation performance by a simple polynomial regression. The novel part of our method is thereby the combination of the multi-task network, the (online) depth evaluation and the regression to a performance prediction framework, which is potentially generalizable to other computer vision tasks and works on a single-image basis (cf.…”

Section: Original Contributionmentioning

confidence: 99%

“…Following previous approaches [36], [55] we use a mixture of absolute difference and structured similarity (SSIM) difference inside the loss function with a weighting factor of α = 0.85. Also adapting the per-pixel minimum reprojection loss from [4] to handle occlusions between different consecutive frames, we can define the photometric loss as:…”

Section: B Self-supervised Monocular Depth Estimationmentioning

confidence: 99%

See 4 more Smart Citations

Online Performance Prediction of Perception DNNs by Multi-Task Learning With Depth Estimation

Klingner

Fingscheidt

2021

IEEE Trans. Intell. Transport. Syst.

Self Cite

View full text Add to dashboard Cite

Online performance prediction (or: observation) of deep neural networks (DNNs) in highly automated driving presents an unsolved task until now, as most DNNs are evaluated offline requiring datasets with ground truth labels. In practice, however, DNN performance depends on the used camera type, lighting and weather conditions, and on various other kinds of domain shift. Also, the input to DNN-based perception systems can be perturbed by adversarial attacks requiring means to detect these input perturbations. In this work we propose a method to mitigate these problems by a multi-task learning approach with monocular depth estimation as a secondary task, which enables us to predict the DNN's performance for various other (primary) tasks by evaluating only the depth estimation task with a physical depth measurement provided, e.g., by a LiDAR sensor. We show the effectiveness of our method for the primary task of semantic segmentation using various training datasets, test datasets, model architectures, and input perturbations. Our method provides an effective way to predict (observe) the performance of DNNs for semantic segmentation even on a single-image basis and is transferable to other primary DNN-based perception tasks in a straightforward manner.

show abstract

Section: B Multi-task Learningmentioning

confidence: 99%

Section: Performance Prediction Of Neural Networkmentioning

confidence: 99%

Section: Original Contributionmentioning

confidence: 99%

Section: Original Contributionmentioning

confidence: 99%

Section: B Self-supervised Monocular Depth Estimationmentioning

confidence: 99%

See 3 more Smart Citations

Online Performance Prediction of Perception DNNs by Multi-Task Learning With Depth Estimation

Klingner

Fingscheidt

2021

IEEE Trans. Intell. Transport. Syst.

Self Cite

View full text Add to dashboard Cite

show abstract

Robustness Verification of Semantic Segmentation Neural Networks Using Relaxed Reachability

Tran

Pal

Musau

et al. 2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

This paper introduces robustness verification for semantic segmentation neural networks (in short, semantic segmentation networks [SSNs]), building on and extending recent approaches for robustness verification of image classification neural networks. Despite recent progress in developing verification methods for specifications such as local adversarial robustness in deep neural networks (DNNs) in terms of scalability, precision, and applicability to different network architectures, layers, and activation functions, robustness verification of semantic segmentation has not yet been considered. We address this limitation by developing and applying new robustness analysis methods for several segmentation neural network architectures, specifically by addressing reachability analysis of up-sampling layers, such as transposed convolution and dilated convolution. We consider several definitions of robustness for segmentation, such as the percentage of pixels in the output that can be proven robust under different adversarial perturbations, and a robust variant of intersection-over-union (IoU), the typical performance evaluation measure for segmentation tasks. Our approach is based on a new relaxed reachability method, allowing users to select the percentage of a number of linear programming problems (LPs) to solve when constructing the reachable set, through a relaxation factor percentage. The approach is implemented within NNV, then applied and evaluated on segmentation datasets, such as a multi-digit variant of MNIST known as M2NIST. Thorough experiments show that by using transposed convolution for up-sampling and average-pooling for down-sampling, combined with minimizing the number of ReLU layers in the SSNs, we can obtain SSNs with not only high accuracy (IoU), but also that are more robust to adversarial attacks and amenable to verification. Additionally, using our new relaxed reachability method, we can significantly reduce the verification time for neural networks whose ReLU layers dominate the total analysis time, even in classification tasks.

show abstract

Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety

Houben

Abrecht

Akila

et al. 2022

Deep Neural Networks and Data for Automated Driving

Self Cite

View full text Add to dashboard Cite

Deployment of modern data-driven machine learning methods, most often realized by deep neural networks (DNNs), in safety-critical applications such as health care, industrial plant control, or autonomous driving is highly challenging due to numerous model-inherent shortcomings. These shortcomings are diverse and range from a lack of generalization over insufficient interpretability and implausible predictions to directed attacks by means of malicious inputs. Cyber-physical systems employing DNNs are therefore likely to suffer from so-called safety concerns, properties that preclude their deployment as no argument or experimental setup can help to assess the remaining risk. In recent years, an abundance of state-of-the-art techniques aiming to address these safety concerns has emerged. This chapter provides a structured and broad overview of them. We first identify categories of insufficiencies to then describe research activities aiming at their detection, quantification, or mitigation. Our work addresses machine learning experts and safety engineers alike: The former ones might profit from the broad range of machine learning topics covered and discussions on limitations of recent methods. The latter ones might gain insights into the specifics of modern machine learning methods. We hope that this contribution fuels discussions on desiderata for machine learning systems and strategies on how to help to advance existing approaches accordingly.

show abstract

Improved Noise and Attack Robustness for Semantic Segmentation by Using Multi-Task Training with Self-Supervised Depth Estimation

Cited by 45 publications

References 51 publications

Online Performance Prediction of Perception DNNs by Multi-Task Learning With Depth Estimation

Online Performance Prediction of Perception DNNs by Multi-Task Learning With Depth Estimation

Robustness Verification of Semantic Segmentation Neural Networks Using Relaxed Reachability

Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety

Contact Info

Product

Resources

About