DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation

Abstract

As acquiring pixel-wise annotations of real-world images for semantic segmentation is a costly process, a model can instead be trained with more accessible synthetic data and adapted to real images without requiring their annotations. This process is studied in unsupervised domain adaptation (UDA). Even though a large number of methods propose new adaptation strategies, they are mostly based on outdated network architectures. As the influence of recent network architectures has not been systematically studied, we first benchmark different network architectures for UDA and newly reveal the potential of Transformers for UDA semantic segmentation. Based on the findings, we propose a novel UDA method, DAFormer. The network architecture of DAFormer consists of a Transformer encoder and a multilevel context-aware feature fusion decoder. It is enabled by three simple but crucial training strategies to stabilize the training and to avoid overfitting to the source domain: While (1) Rare Class Sampling on the source domain improves the quality of the pseudo-labels by mitigating the confirmation bias of self-training toward common classes, (2) a Thing-Class ImageNet Feature Distance and (3) a learning rate warmup promote feature transfer from Ima-geNet pretraining. DAFormer represents a major advance in UDA. It improves the state of the art by 10.8 mIoU for GTA→Cityscapes and 5.4 mIoU for Synthia→Cityscapes and enables learning even difficult classes such as train, bus, and truck well. The implementation is available at https://github.com/lhoyer/DAFormer.

Exploiting Instance-based Mixed Sampling via Auxiliary Source Domain Supervision for Domain-adaptive Action Detection

“…Adversarial training follows a GAN framework [23,25] to aligns the source and target domains feature distributions at input [24,28], output [77,79], patch [13], or feature level [29,77]. In self-training, the supervision for target domain comes from pseudo-labels [38] which can be computed offline [65,86,93,94] or online [76,80,31]. Consistency regularization [70,75] or label prototypes [88] formulated on CDMS [76,91] or data augmentation [3,14,49] DA action recognition and detection.…”

“…We don't consider multi-label action classes, i.e., those action categories in which action instances could have more than one class label. A rare-class sampling [31] could be helpful to generate more diversified training samples for the under-represented classes. precision of class take a photo improves over 10%.…”

Learning from Future: A Novel Self-Training Framework for Semantic Segmentation

“…By leveraging the model itself to generate pseudo-labels on unlabeled data, self-training together with tailored strategies such as consistency regularization [77,3], cross-domain mixup [55,78], contrastive learning [31,41,79,36], pseudo-label refine [63,73,76], auxiliary tasks [60,62] and class balanced training [35] achieves excellent performance. Recently, Hoyer et al [29] empirically proved that the transformer architecture [67] is more robust to domain shift than CNN.…”

“…Improving the labeling efficiency of deep learning algorithms is vital in practice since acquiring high-quality annotations could consume great effort. Self-training (ST) offers a promising solution to alleviate this issue by learning with limited labeled data and large-scale unlabeled data [57,29]. The key thought is to learn a model on labeled samples and use it to generate pseudo-labels for unlabeled samples to teach the model itself.…”

“…Self-training provides a unified solution and achieves state-of-the-art performance on both settings [29,64]. One of the most common and widely used forms of self-training in semantic segmentation is a variant of mean teacher, which is shown in Fig.…”