CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows

Dong, Xiaoyi; Bao, Jianmin; Chen, Dongdong; Zhang, Weimin; Yu, Nenghai; Yuan, Lu; Chen, Dong; Guo, Baining

doi:10.1109/cvpr52688.2022.01181

Cited by 512 publications

(230 citation statements)

References 29 publications

Supporting

Mentioning

164

Contrasting

Order By: Relevance

“…Backbone mIoU FCN [1] CNN 29.4 RefineNet [35] ResNet-152 40.7 UperNet [17] ResNet-50 41.2 UperNet+Conv [14] ConvNeXt-XL 54.0 UperNet+DeAtt [25] DAT-B 49.4 DeepLabv3++ [36] Xception-65 45.7 Auto-DeepLab [37] NAS 44.0 OCR [38] HRNetV2-W48 45.7 MaskFormer [41] ResNet-50 44.5 MaskFormer+FaPN [11] Swin-L 55.2 SegFormer [21] MiT-B5 51.8 HRViT [43] HRViT-b3 50.2 BEiT [44] Transformer 47.7 CSWin [45] CSWin-L 54.0 Mask2Former [6] Swin-L 56. Dilated-ResNet-101 80.2 RefineNet [35] ResNet-101 73.6 DeepLabv3++ [36] Xception-65 82.1 Auto-DeepLab [37] NAS 80.3 OCR [38] HRNetV2-W48 83.6 MDEQ [39] MDEQ 80.3 SynBoost [40] VGG-16 & CNN 83.5 MaskFormer+FaPN [11] ResNet-101 80.1 SML [42] ResNet-101 80.3 SegFormer [21] MiT-B5 84.0 HRViT [43] HRViT-b3 83.2 HSB-Net [13] ResNet-34 73.1 Mask2Former [6] Swin-L 83.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Feature High-Boosting for Semantic Segmentation

Kim

Lee

et al. 2022

IEEE Access

View full text Add to dashboard Cite

Semantic segmentation has been actively studied with the great success of deep learning. Various network architectures for this task have been introduced over last few years, and the current stateof-the-art methods have shown significant advances enough to be applied to practical applications. However, previous methods often fail to segment small objects due to the limit on the spatial resolution of the feature map encoded through the deep network architecture. In this paper, we propose a simple yet effective module for semantic segmentation. Its key components include feature high-boosting, which efficiently highlights boundaries of small objects by simple operations, i.e., sum of the original feature and the corresponding residual, in the latent space. This is fairly desirable to guide network parameters to focus on features of small objects. Experimental results on various benchmarks show that the proposed method successfully improves the performance of semantic segmentation regardless of backbone networks. In particular, the proposed method with the transformer-based decoder achieves 56.4 mIoU on the ADE20K dataset.INDEX TERMS Semantic segmentation, deep learning, feature high-boosting, boundaries of small objects, latent space.

show abstract

Section: Methodsmentioning

confidence: 99%

“…and CSWin [45], based on the mean IoU (mIoU) metric, which describes the intersection-over-union between the predicted result and the ground truth. The corresponding results on ADE20K and Cityscape datasets are shown in Tables 1 and 2, respectively.…”

Section: Performance Evaluationmentioning

confidence: 99%

Feature High-Boosting for Semantic Segmentation

Kim

Lee

et al. 2022

IEEE Access

View full text Add to dashboard Cite

show abstract

“…As shown in Fig. 1, first, the CSwin Transformer [5] encoder takes a RGB image as input, and split the image into nonoverlapping patches as ªtokensº. The whole encoder consists of four stages to produce hierarchical representations (i.e., ST 1 , ST 2 , ST 3 , and ST 4 ).…”

Section: Network Architecturementioning

confidence: 99%

“…(2) We substitute the encoder of Maskformer from Swin transformer to CSwin transformer [5]. The latter introduces the cross-shaped window self-attention mechanism that computing self-attention in the horizontal and vertical stripes in parallel to enlarges the attention area of each token without the increasing of computational complexity.…”

Section: Introductionmentioning

confidence: 99%

Maskformer with Improved Encoder-Decoder Module for Semantic Segmentation of Fine-Resolution Remote Sensing Images

Yang

Wang

et al. 2022

2022 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

In 2021, the Transformer based models have demonstrated extraordinary achievement in the field of computer vision. Among which, Maskformer, a Transformer based model adopting the mask classification method, is an outstanding model in both semantic segmentation and instance segmentation. Considering the specific characteristics of semantic segmentation of remote sensing images (RSIs), we design CADA-MaskFormer(a Mask classification-based model with Cross-shaped window self-Attention and Densely connected feature Aggregation) based on Maskformer by improving its encoder and pixel decoder. Concretely, the mask classification that generates one or even more masks for specific category to perform the elaborate segmentation is especially suitable for handling the characteristic of large within-class and small between-class variance of RSIs. Furthermore, we apply the Cross-Shaped Window self-attention mechanism to model the long-range context information contained in RSIs at maximum extent without the increasing of computational complexity. In addition, the Densely Connected Feature Aggregation Module (DCFAM) is used as the pixel decoder to incorporate multi-level feature maps from the encoder to get a finer semantic segmentation map. Extensive experiments conducted on two remotely sensed semantic segmentation datasets Potsdam and Vaihingen achieves 91.88% and 91.01% in OA index respectively, outperforming most of competitive models designed for RSIs. The code is available from https://github.com/lqwrl542293/JL-Yang_ CV/tree/master/CADA_Maskformer

show abstract

“…To reduce the number of tokens, two camps of approaches are proposed. The first is to apply the concepts of hierarchical convolutional neural nets to downsample tokens using various pooling methods (Liu et al, 2021;Heo et al, 2021;Dong et al, 2022). The other attempts to measure the significance scores among the tokens and drop or prune tokens accordingly (Goyal et al, 2020;Rao et al, 2021;Marin et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

CAST: Concurrent Recognition and Segmentation with Adaptive Segment Tokens

Ke¹,

Hwang²,

Yu³

2022

Preprint

View full text Add to dashboard Cite

Recognizing an image and segmenting it into coherent regions are often treated as separate tasks. Human vision, however, has a general sense of segmentation hierarchy before recognition occurs. We are thus inspired to learn image recognition with hierarchical image segmentation based entirely on unlabeled images. Our insight is to learn fine-to-coarse features concurrently at superpixels, segments, and full image levels, enforcing consistency and goodness of feature induced segmentations while maximizing discrimination among image instances. Our model innovates vision transformers on three aspects. 1) We use adaptive segment tokens instead of fixed-shape patch tokens. 2) We create a token hierarchy by inserting graph pooling between transformer blocks, naturally producing consistent multi-scale segmentations while increasing the segment size and reducing the number of tokens. 3) We produce hierarchical image segmentation for free while training for recognition by maximizing image-wise discrimination. Our work delivers the first concurrent recognition and hierarchical segmentation model without any supervision. Validated on ImageNet and PASCAL VOC, it achieves better recognition and segmentation with higher computational efficiency.

show abstract

CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows

Cited by 512 publications

References 29 publications

Feature High-Boosting for Semantic Segmentation

Feature High-Boosting for Semantic Segmentation

Maskformer with Improved Encoder-Decoder Module for Semantic Segmentation of Fine-Resolution Remote Sensing Images

CAST: Concurrent Recognition and Segmentation with Adaptive Segment Tokens

Contact Info

Product

Resources

About