Vision Transformer with Deformable Attention

Xia, Zhuofan; Pan, Xuran; Song, Sejun; Li, Li Erran; Huang, Gao

doi:10.1109/cvpr52688.2022.00475

Cited by 235 publications

(100 citation statements)

References 94 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our DGMN2 is related to DAT [76] and Deformable DETR [91], but has key differences: When calculating attention, DAT [76] learns a offset for each point on feature map and use the deformed points as the key and value. In contrast, our DGMN2 samples K nodes of key and value and calculate the attention using sampled key and value.…”

Section: Discussionmentioning

confidence: 99%

Dynamic Graph Message Passing Networks for Visual Recognition

Zhang

Chen

Arnab

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Modelling long-range dependencies is critical for scene understanding tasks in computer vision. Although convolution neural networks (CNNs) have excelled in many vision tasks, they are still limited in capturing long-range structured relationships as they typically consist of layers of local kernels. A fully-connected graph, such as the self-attention operation in Transformers, is beneficial for such modelling, however, its computational overhead is prohibitive. In this paper, we propose a dynamic graph message passing network, that significantly reduces the computational complexity compared to related works modelling a fully-connected graph. This is achieved by adaptively sampling nodes in the graph, conditioned on the input, for message passing. Based on the sampled nodes, we dynamically predict node-dependent filter weights and the affinity matrix for propagating information between them. This formulation allows us to design a self-attention module, and more importantly a new Transformer-based backbone network, that we use for both image classification pretraining, and for addressing various downstream tasks (e.g. object detection, instance and semantic segmentation). Using this model, we show significant improvements with respect to strong, state-of-the-art baselines on four different tasks. Our approach also outperforms fully-connected graphs while using substantially fewer floating-point operations and parameters. Code and models will be made publicly available at https://github.com/fudan-zvg/DGMN2.

show abstract

Section: Discussionmentioning

confidence: 99%

Dynamic Graph Message Passing Networks for Visual Recognition

Zhang

Chen

Arnab

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…Backbone mIoU FCN [1] CNN 29.4 RefineNet [35] ResNet-152 40.7 UperNet [17] ResNet-50 41.2 UperNet+Conv [14] ConvNeXt-XL 54.0 UperNet+DeAtt [25] DAT-B 49.4 DeepLabv3++ [36] Xception-65 45.7 Auto-DeepLab [37] NAS 44.0 OCR [38] HRNetV2-W48 45.7 MaskFormer [41] ResNet-50 44.5 MaskFormer+FaPN [11] Swin-L 55.2 SegFormer [21] MiT-B5 51.8 HRViT [43] HRViT-b3 50.2 BEiT [44] Transformer 47.7 CSWin [45] CSWin-L 54.0 Mask2Former [6] Swin-L 56. Dilated-ResNet-101 80.2 RefineNet [35] ResNet-101 73.6 DeepLabv3++ [36] Xception-65 82.1 Auto-DeepLab [37] NAS 80.3 OCR [38] HRNetV2-W48 83.6 MDEQ [39] MDEQ 80.3 SynBoost [40] VGG-16 & CNN 83.5 MaskFormer+FaPN [11] ResNet-101 80.1 SML [42] ResNet-101 80.3 SegFormer [21] MiT-B5 84.0 HRViT [43] HRViT-b3 83.2 HSB-Net [13] ResNet-34 73.1 Mask2Former [6] Swin-L 83.…”

Section: Methodsmentioning

confidence: 99%

“…Yu et al [24] proposed to replace the self-attention module with a simple spatial pooling operator for the transformer and showed the competitive performance while significantly reducing the processing complexity. In [25], authors proposed to design a deformable self-attention module, which makes the attention block to concentrate on relevant regions in a data-aware way. However, it is difficult to maintain the highresolution feature map for the transformer-based architecture due to the constraint in terms of computational burden and memory limitation.…”

Section: Related Workmentioning

confidence: 99%

“…Quantitative evaluation. To demonstrate the performance of the proposed method, we compare ours with rep- resentative approaches for semantic segmentation, e.g., FCN [1], PSPNet [7], RefineNet [35], UperNet [17], Uper-Net+Conv [14], UperNet+DeAtt [25], DeepLabv3++ [36], Auto-DeepLab [37], OCR [38], MDEQ [39], SynBoost [40], MaskFormer [41], FaPN [11], SML [42], SegFormer [21],…”

Section: Performance Evaluationmentioning

confidence: 99%

See 1 more Smart Citation

Feature High-Boosting for Semantic Segmentation

Kim

Lee

et al. 2022

IEEE Access

View full text Add to dashboard Cite

Semantic segmentation has been actively studied with the great success of deep learning. Various network architectures for this task have been introduced over last few years, and the current stateof-the-art methods have shown significant advances enough to be applied to practical applications. However, previous methods often fail to segment small objects due to the limit on the spatial resolution of the feature map encoded through the deep network architecture. In this paper, we propose a simple yet effective module for semantic segmentation. Its key components include feature high-boosting, which efficiently highlights boundaries of small objects by simple operations, i.e., sum of the original feature and the corresponding residual, in the latent space. This is fairly desirable to guide network parameters to focus on features of small objects. Experimental results on various benchmarks show that the proposed method successfully improves the performance of semantic segmentation regardless of backbone networks. In particular, the proposed method with the transformer-based decoder achieves 56.4 mIoU on the ADE20K dataset.INDEX TERMS Semantic segmentation, deep learning, feature high-boosting, boundaries of small objects, latent space.

show abstract

“…Large-scale pre-training. Benefited from the development of Transformer in both vision [35,63,36] and language [54] tasks, large-scale pre-training framework has received wide concerns in recent years and shown promising results in the field of computer vision and natural language processing. GPT [39] is one of the pioneer works for language pre-training which optimizes the probability of output based on previous words in the sequence.…”

Section: Related Workmentioning

confidence: 99%

Contrastive Language-Image Pre-Training with Knowledge Graphs

Pan¹,

Tian-zhu²,

Han³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent years have witnessed the fast development of large-scale pre-training frameworks that can extract multi-modal representations in a unified form and achieve promising performances when transferred to downstream tasks. Nevertheless, existing approaches mainly focus on pre-training with simple image-text pairs, while neglecting the semantic connections between concepts from different modalities. In this paper, we propose a knowledge-based pre-training framework, dubbed Knowledge-CLIP, which injects semantic information into the widely used CLIP model [38]. Through introducing knowledge-based objectives in the pre-training process and utilizing different types of knowledge graphs as training data, our model can semantically align the representations in vision and language with higher quality, and enhance the reasoning ability across scenarios and modalities. Extensive experiments on various vision-language downstream tasks demonstrate the effectiveness of Knowledge-CLIP compared with the original CLIP and competitive baselines.

show abstract

Vision Transformer with Deformable Attention

Cited by 235 publications

References 94 publications

Dynamic Graph Message Passing Networks for Visual Recognition

Dynamic Graph Message Passing Networks for Visual Recognition

Feature High-Boosting for Semantic Segmentation

Contrastive Language-Image Pre-Training with Knowledge Graphs

Contact Info

Product

Resources

About