TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation

Zhang, Yundong; Liu, Huiye; Hu, Qiang

doi:10.48550/arxiv.2102.08005

Cited by 40 publications

(48 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The overall architecture of TransUNet is similar to that of U-Net [27], where convnets act as feature extractors and transformers help encode the global context. In fact, a major feature of TransUNet and most of its followers [4,5,32,41] is to treat convnets as main bodies, on top of which transformers are further applied to capture long-term dependencies. However, such characteristic may cause a problem: advantages of transformers are not fully exploited.…”

Section: Introductionmentioning

confidence: 99%

nnFormer: Interleaved Transformer for Volumetric Segmentation

Zhou¹,

Guo²,

Zhang³

et al. 2021

Preprint

104

View full text Add to dashboard Cite

Transformers, the default model of choices in natural language processing, have drawn scant attention from the medical imaging community. Given the ability to exploit long-term dependencies, transformers are promising to help atypical convolutional neural networks (convnets) to overcome its inherent shortcomings of spatial inductive bias. However, most of recently proposed transformer-based segmentation approaches simply treated transformers as assisted modules to help encode global context into convolutional representations without investigating how to optimally combine self-attention (i.e., the core of transformers) with convolution. To address this issue, in this paper, we introduce nnFormer (i.e., not-another transFormer), a powerful segmentation model with an interleaved architecture based on empirical combination of self-attention and convolution. In practice, nnFormer learns volumetric representations from 3D local volumes. Compared to the naive voxel-level self-attention implementation, such volume-based operations help to reduce the computational complexity by approximate 98% and 99.5% on Synapse and ACDC datasets, respectively. In comparison to prior-art network configurations, nnFormer achieves tremendous improvements over previous transformer-based methods on two commonly used datasets Synapse and ACDC. For instance, nnFormer outperforms Swin-UNet by over 7 percents on Synapse. Even when compared to nnUNet, currently the best performing fully-convolutional medical segmentation network, nnFormer still provides slightly better performance on Synapse and ACDC. Codes and models are available at https://github.com/282857341/nnFormer.

show abstract

Section: Introductionmentioning

confidence: 99%

nnFormer: Interleaved Transformer for Volumetric Segmentation

Zhou¹,

Guo²,

Zhang³

et al. 2021

Preprint

104

View full text Add to dashboard Cite

show abstract

“…With skip-connection incorporated, TransUnet sets new records (at the time of publication) on synapse multi-organ segmentation dataset [156] and automated cardiac diagnosis challenge (ACDC) [155]. In other work, Zhang et al propose TransFuse [157] to effectively fuse features from the Transformer and CNN layers via BiFusion module. The BiFusion module leverages the self-attention and multi-modal fusion mechanism to selectively fuse the features.…”

Section: Hybrid Architecturesmentioning

confidence: 99%

“…Extensive evaluation of Trans-Fuse on multiple modalities (2D and 3D), including Polyp segmentation, skin lesion segmentation, Hip segmentation, and prostate segmentation, demonstrate its efficacy. Both TransUNet [96] and TransFuse [157] require pre-training on ImageNet dataset [158] to effectively learn the positional encoding of the images. To learn this positional bias without any pre-training, Valanarasu et al [128] propose a modified gated axial attention layer [159] that works well on small medical image segmentation datasets.…”

Section: Hybrid Architecturesmentioning

confidence: 99%

Transformers in Medical Imaging: A Survey

Shamshad¹,

Khan²,

Zamir³

et al. 2022

Preprint

View full text Add to dashboard Cite

Following unprecedented success on the natural language tasks, Transformers have been successfully applied to several computer vision problems, achieving state-of-the-art results and prompting researchers to reconsider the supremacy of convolutional neural networks (CNNs) as de facto operators. Capitalizing on these advances in computer vision, the medical imaging field has also witnessed growing interest for Transformers that can capture global context compared to CNNs with local receptive fields. Inspired from this transition, in this survey, we attempt to provide a comprehensive review of the applications of Transformers in medical imaging covering various aspects, ranging from recently proposed architectural designs to unsolved issues. Specifically, we survey the use of Transformers in medical image segmentation, detection, classification, reconstruction, synthesis, registration, clinical report generation, and other tasks. In particular, for each of these applications, we develop taxonomy, identify application-specific challenges as well as provide insights to solve them, and highlight recent trends. Further, we provide a critical discussion of the field's current state as a whole, including the identification of key challenges, open problems, and outlining promising future directions. We hope this survey will ignite further interest in the community and provide researchers with an up-to-date reference regarding applications of Transformer models in medical imaging. Finally, to cope with the rapid development in this field, we intend to regularly update the relevant latest papers and their open-source implementations at https://github.com/fahadshamshad/awesome-transformers-in-medical-imaging.

show abstract

“…TransUNet [34] employs ViT as encoder with large-scale pre-training for medical image segmentation. TransFuse [35] combines Transformer and CNN in parallel to effectively capture global dependencies and spatial details in a much shallower manner. Besides, MedT [36] proposes a Gated Axial-Attention model which applies an additional control mechanism in the self-attention module to extend existing architectures.…”

Section: B Transformer-based Approaches 1) Transformer For Various Vi...mentioning

confidence: 99%

TransBTSV2: Towards Better and More Efficient Volumetric Segmentation of Medical Images

Li¹,

Wang²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Transformer, benefiting from global (longrange) information modeling using self-attention mechanism, has been successful in natural language processing and computer vision recently. Convolutional Neural Networks, capable of capturing local features, are unable to model explicit long-distance dependencies from global feature space. However, both local and global features are crucial for dense prediction tasks, especially for 3D medical image segmentation. In this paper, we exploit Transformer in 3D CNN for 3D medical image volumetric segmentation and propose a novel network named TransBTSV2 based on the encoder-decoder structure. Different from our original TransBTS [1], the proposed TransBTSV2 is not limited to brain tumor segmentation (BTS) but focuses on general medical image segmentation, providing a strong and efficient 3D baseline for volumetric segmentation of medical images. As a hybrid CNN-Transformer architecture, TransBTSV2 can achieve accurate segmentation of medical images without any pre-training. With the proposed insight to redesign the internal structure of Transformer and the introduced Deformable Bottleneck Module, a highly efficient architecture is achieved with superior performance. Extensive experimental results on four medical image datasets (BraTS 2019, BraTS 2020, LiTS 2017 and KiTS 2019) demonstrate that TransBTSV2 achieves comparable or better results as compared to the state-of-the-art methods for the segmentation of brain tumor, liver tumor as well as kidney tumor. Code is available at https://github.com/ Wenxuan-1119/TransBTS.

show abstract

TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation

Cited by 40 publications

References 36 publications

nnFormer: Interleaved Transformer for Volumetric Segmentation

nnFormer: Interleaved Transformer for Volumetric Segmentation

Transformers in Medical Imaging: A Survey

TransBTSV2: Towards Better and More Efficient Volumetric Segmentation of Medical Images

Contact Info

Product

Resources

About