Continual Unsupervised Domain Adaptation for Semantic Segmentation by Online Frequency Domain Style Transfer

Termöhlen, Jan-Aike; Klingner, Marvin; Brettin, Leon J.; Schmidt, Nico M.; Fingscheidt, Tim

doi:10.1109/itsc48978.2021.9564566

Cited by 18 publications

(13 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most methods in this category store information about a specific domain's style in order to transform source images into the styles of the target domains during training. Recent work achieves this by storing low-frequency components of the images for every domain [14], by capturing the style of the domain with generative models [15], [16] or by using a domain-specific memory to mitigate forgetting [17]. Supervised domain-incremental learning is rarely considered [11].…”

Section: A Continual Semantic Segmentationmentioning

confidence: 99%

“…Methods in this category work mostly by storing information about the style of the specific domains, so that during training the source images can be transferred into the styles of the different target domains. This can be achieved by storing low-frequency components of the domains [53] or by capturing the style using generative models [36,60]. Other recent work proposes to use a target-specific memory for every domain to mitigate forgetting [28].…”

Section: Continual Unsupervised Domain Adaptationmentioning

confidence: 99%

“…This phenomenon is usually referred to as catastrophic forgetting and is a fundamental challenge when training a neural network on a continuous stream of data. Recently proposed methods mostly mitigate this challenge by replaying data from previous domains, re-estimating statistics or even in an unsupervised manner by transferring training images in the style of the novel domain [36,53,60]. The focus of our work is to study how the internal representations of se-mantic segmentation models are affected during domainincremental learning and how efficient feature reuse can mitigate forgetting without explicit replay of the previous domain.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Causes of Catastrophic Forgetting in Class-Incremental Semantic Segmentation

Kalb¹,

Beyerer

2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Research in the field of Continual Semantic Segmentation is mainly investigating novel learning algorithms to overcome catastrophic forgetting of neural networks. Most recent publications have focused on improving learning algorithms without distinguishing effects caused by the choice of neural architecture. Therefore, we study how the choice of neural network architecture affects catastrophic forgetting in class-and domainincremental semantic segmentation. Specifically, we compare the well-researched CNNs to recently proposed Transformers and Hybrid architectures, as well as the impact of the choice of novel normalization layers and different decoder heads. We find that traditional CNNs like ResNet have high plasticity but low stability, while transformer architectures are much more stable. When the inductive biases of CNN architectures are combined with transformers in hybrid architectures, it leads to higher plasticity and stability. The stability of these models can be explained by their ability to learn general features that are robust against distribution shifts. Experiments with different normalization layers show that Continual Normalization achieves the best trade-off in terms of adaptability and stability of the model. In the class-incremental setting, the choice of the normalization layer has much less impact. Our experiments suggest that the right choice of architecture can significantly reduce forgetting even with naive fine-tuning and confirm that for real-world applications, the architecture is an important factor in designing a continual learning model.

show abstract

Section: A Continual Semantic Segmentationmentioning

confidence: 99%

Section: Continual Unsupervised Domain Adaptationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Causes of Catastrophic Forgetting in Class-Incremental Semantic Segmentation

Kalb¹,

Beyerer

2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…This network replays source domain knowledge to the network during adaptation. Moreover, Stan et al [47] and Termöhlen et al [48] learn a source domain distribution, which is aligned with the target domain distribution during adaptation. These approaches do not make use of source data during the UDA.…”

Section: Uda Without Source Datamentioning

confidence: 99%

Continual BatchNorm Adaptation (CBNA) for Semantic Segmentation

Klingner

Ayache

Fingscheidt

2022

IEEE Trans. Intell. Transport. Syst.

Self Cite

View full text Add to dashboard Cite

Environment perception in autonomous driving vehicles often heavily relies on deep neural networks (DNNs), which are subject to domain shifts, leading to a significantly decreased performance during DNN deployment. Usually, this problem is addressed by unsupervised domain adaptation (UDA) approaches trained either simultaneously on source and target domain datasets or even source-free only on target data in an offline fashion. In this work, we further expand a source-free UDA approach to a continual and therefore online-capable UDA on a single-image basis for semantic segmentation. Accordingly, our method only requires the pre-trained model from the supplier (trained in the source domain) and the current (unlabeled target domain) camera image. Our method Continual BatchNorm Adaptation (CBNA) modifies the source domain statistics in the batch normalization layers, using target domain images in an unsupervised fashion, which yields consistent performance improvements during inference. Thereby, in contrast to existing works, our approach can be applied to improve a DNN continuously on a single-image basis during deployment without access to source data, without algorithmic delay, and nearly without computational overhead. We show the consistent effectiveness of our method across a wide variety of source/target domain settings for semantic segmentation. Code is available at https://github.com/ifnspaml/CBNA

show abstract

Section: Uda Without Source Datamentioning

confidence: 99%

Continual BatchNorm Adaptation (CBNA) for Semantic Segmentation

Klingner¹,

Ayache²,

Fingscheidt³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Environment perception in autonomous driving vehicles often heavily relies on deep neural networks (DNNs), which are subject to domain shifts, leading to a significantly decreased performance during DNN deployment. Usually, this problem is addressed by unsupervised domain adaptation (UDA) approaches trained either simultaneously on source and target domain datasets or even source-free only on target data in an offline fashion. In this work, we further expand a source-free UDA approach to a continual and therefore online-capable UDA on a single-image basis for semantic segmentation. Accordingly, our method only requires the pre-trained model from the supplier (trained in the source domain) and the current (unlabeled target domain) camera image. Our method Continual BatchNorm Adaptation (CBNA) modifies the source domain statistics in the batch normalization layers, using target domain images in an unsupervised fashion, which yields consistent performance improvements during inference. Thereby, in contrast to existing works, our approach can be applied to improve a DNN continuously on a single-image basis during deployment without access to source data, without algorithmic delay, and nearly without computational overhead. We show the consistent effectiveness of our method across a wide variety of source/target domain settings for semantic segmentation. As part of this work, our code will be made publicly available. 1

show abstract

Continual Unsupervised Domain Adaptation for Semantic Segmentation by Online Frequency Domain Style Transfer

Cited by 18 publications

References 40 publications

Causes of Catastrophic Forgetting in Class-Incremental Semantic Segmentation

Causes of Catastrophic Forgetting in Class-Incremental Semantic Segmentation

Continual BatchNorm Adaptation (CBNA) for Semantic Segmentation

Continual BatchNorm Adaptation (CBNA) for Semantic Segmentation

Contact Info

Product

Resources

About