A survey on attention mechanisms for medical applications: are we moving towards better algorithms?

Abstract

The increasing popularity of attention mechanisms in deep learning algorithms for computer vision and natural language processing made these models attractive to other research domains. In healthcare, there is a strong need for tools that may improve the routines of the clinicians and the patients. Naturally, the use of attention-based algorithms for medical applications occurred smoothly. However, being healthcare a domain that depends on high-stake decisions, the scientific community must ponder if these high-performing algorithms fit the needs of medical applications. With this motto, this paper extensively reviews the use of attention mechanisms in machine learning (including Transformers) for several medical applications. This work distinguishes itself from its predecessors by proposing a critical analysis of the claims and potentialities of attention mechanisms presented in the literature through an experimental case study on medical image classification with three different use cases. These experiments focus on the integrating process of attention mechanisms into established deep learning architectures, the analysis of their predictive power, and a visual assessment of their saliency maps generated by post-hoc explanation methods. This paper concludes with a critical analysis of the claims and potentialities presented in the literature about attention mechanisms and proposes future research lines in medical applications that may benefit from these frameworks.

LARF: Two-Level Attention-Based Random Forests with a Mixture of Contamination Models

“…Another improvement of the attention-based models is to use the self-attention which was proposed in [16] as a crucial component of neural networks called Transformers. The self-attention models have also been studied in surveys [4,[24][25][26][27][28]. This is only a small part of all the works devoted to attention and self-attention mechanisms.…”