A Survey of Transformers

Lin, Tianyang; Wang, Yuxin; Liu, Xiangyang; Qiu, Xipeng

doi:10.48550/arxiv.2106.04554

Cited by 75 publications

(96 citation statements)

References 112 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Below, we briefly describe the core components behind the success of ViTs that are self-attention and multi-head selfattention. For a more in-depth analysis of numerous ViT architectures and applications, we refer interesting readers to the recent relevant survey papers [23], [81]- [84].…”

Section: Transformersmentioning

confidence: 99%

Transformers in Medical Imaging: A Survey

Shamshad¹,

Khan²,

Zamir³

et al. 2022

Preprint

View full text Add to dashboard Cite

Following unprecedented success on the natural language tasks, Transformers have been successfully applied to several computer vision problems, achieving state-of-the-art results and prompting researchers to reconsider the supremacy of convolutional neural networks (CNNs) as de facto operators. Capitalizing on these advances in computer vision, the medical imaging field has also witnessed growing interest for Transformers that can capture global context compared to CNNs with local receptive fields. Inspired from this transition, in this survey, we attempt to provide a comprehensive review of the applications of Transformers in medical imaging covering various aspects, ranging from recently proposed architectural designs to unsolved issues. Specifically, we survey the use of Transformers in medical image segmentation, detection, classification, reconstruction, synthesis, registration, clinical report generation, and other tasks. In particular, for each of these applications, we develop taxonomy, identify application-specific challenges as well as provide insights to solve them, and highlight recent trends. Further, we provide a critical discussion of the field's current state as a whole, including the identification of key challenges, open problems, and outlining promising future directions. We hope this survey will ignite further interest in the community and provide researchers with an up-to-date reference regarding applications of Transformer models in medical imaging. Finally, to cope with the rapid development in this field, we intend to regularly update the relevant latest papers and their open-source implementations at https://github.com/fahadshamshad/awesome-transformers-in-medical-imaging.

show abstract

Section: Transformersmentioning

confidence: 99%

Transformers in Medical Imaging: A Survey

Shamshad¹,

Khan²,

Zamir³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Many attention-based models have been developed to improve the performance of classification and regression algorithms. Surveys of various attention-based models are available in [2,3,4,5,17,1].…”

Section: Related Workmentioning

confidence: 99%

“…Many attention-based models are also applied to the computer vision area, including image-based analysis, visual question answering, etc. Detailed surveys and reviews of attention, its forms, properties and applications can be found in [2,3,4,5,1].…”

Section: Introductionmentioning

confidence: 99%

Attention-based Random Forest and Contamination Model

Utkin¹,

Konstantinov²

2022

Preprint

View full text Add to dashboard Cite

A new approach called ABRF (the attention-based random forest) and its modifications for applying the attention mechanism to the random forest (RF) for regression and classification are proposed. The main idea behind the proposed ABRF models is to assign attention weights with trainable parameters to decision trees in a specific way. The weights depend on the distance between an instance, which falls into a corresponding leaf of a tree, and instances, which fall in the same leaf. This idea stems from representation of the Nadaraya-Watson kernel regression in the form of a RF. Three modifications of the general approach are proposed. The first one is based on applying the Huber's contamination model and on computing the attention weights by solving quadratic or linear optimization problems. The second and the third modifications use the gradient-based algorithms for computing trainable parameters. Numerical experiments with various regression and classification datasets illustrate the proposed method.

show abstract

“…There are many other variables in the robustness landscape such as embedding technique, robustness metrics, and robustness techniques which were not covered under their survey. Lin et al [29] presented a research survey on transformers which are pre-trained NLP models. The study is centered around robustness from a model stand point where learning models such as BERT and RoBERTa can contribute to robustness.…”

Section: Related Workmentioning

confidence: 99%

Robust Natural Language Processing: Recent Advances, Challenges, and Future Directions

Omar¹,

Choi²,

Nyang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent natural language processing (NLP) techniques have accomplished high performance on benchmark datasets, primarily due to the significant improvement in the performance of deep learning. The advances in the research community have led to great enhancements in state-of-the-art production systems for NLP tasks, such as virtual assistants, speech recognition, and sentiment analysis. However, such NLP systems still often fail when tested with adversarial attacks. The initial lack of robustness exposed troubling gaps in current models' language understanding capabilities, creating problems when NLP systems are deployed in real life. In this paper, we present a structured overview of NLP robustness research by summarizing the literature in a systemic way across various dimensions. We then take a deep-dive into the various dimensions of robustness, across techniques, metrics, embeddings, and benchmarks. Finally, we argue that robustness should be multi-dimensional, provide insights into current research, identify gaps in the literature to suggest directions worth pursuing to address these gaps.

show abstract

A Survey of Transformers

Cited by 75 publications

References 112 publications

Transformers in Medical Imaging: A Survey

Transformers in Medical Imaging: A Survey

Attention-based Random Forest and Contamination Model

Robust Natural Language Processing: Recent Advances, Challenges, and Future Directions

Contact Info

Product

Resources

About