MEDUSA: Multi-Scale Encoder-Decoder Self-Attention Deep Neural Network Architecture for Medical Image Analysis

Abstract

Medical image analysis continues to hold interesting challenges given the subtle characteristics of certain diseases and the significant overlap in appearance between diseases. In this study, we explore the concept of self-attention for tackling such subtleties in and between diseases. To this end, we introduce, a multi-scale encoder-decoder self-attention (MEDUSA) mechanism tailored for medical image analysis. While self-attention deep convolutional neural network architectures in existing literature center around the notion of multiple isolated lightweight attention mechanisms with limited individual capacities being incorporated at different points in the network architecture, MEDUSA takes a significant departure from this notion by possessing a single, unified self-attention mechanism with significantly higher capacity with multiple attention heads feeding into different scales in the network architecture. To the best of the authors' knowledge, this is the first “single body, multi-scale heads” realization of self-attention and enables explicit global context among selective attention at different levels of representational abstractions while still enabling differing local attention context at individual levels of abstractions. With MEDUSA, we obtain state-of-the-art performance on multiple challenging medical image analysis benchmarks including COVIDx, Radiological Society of North America (RSNA) RICORD, and RSNA Pneumonia Challenge when compared to previous work. Our MEDUSA model is publicly available.

CAGNet: A Multi-Scale Convolutional Attention Method for Glass Detection Based on Transformer

“…This module accepts the following four different feature sizes output by the backbone network: ( , , 256), ( , , 512), ( , , 1024), and ( , , 2048). These features are weighted toward the target area through the multi-head self-attention mechanism of a traditional transformer, thereby focusing on the target feature information [34,35]. Each layer corresponded to the following four output feature sizes: ( , , 256), ( , , 512), ( , , 1024), and ( , , 2048).…”

“…This module accepts the following four different feature sizes output by the backbone network: ( ). These features are weighted toward the target area through the multi-head self-attention mechanism of a traditional transformer, thereby focusing on the target feature information [34,35] ). These features were up-sampled and fused through the corresponding GC layer.…”

“…Directly processing these features could potentially limit the model's performance. To address this, a TH was designed to consolidate these features and mitigate the impact of such discrepancies on the model's efficacy [34,35]. This design aimed to enhance the model's performance by effectively integrating and leveraging features at different levels.…”

“…This approach enhances the model's expressive capacity and The TH described herein shares the same structure as the TH and TCH modules. It applies feature weighting to fused features, thereby minimizing the influence of irrelevant information [34,35]. Subsequently, the features were passed through a feed-forward network (FFN) layer for nonlinear transformation, ensuring that the feature dimensions were aligned with the next layer.…”

A Novel COVID-19 Image Classification Method Based on the Improved Residual Network

“…Then, the network utilized a self-attention mechanism to model non-local interactions and learn rich contextual information to detect complex-shaped lesion regions, which improved recognition efficiency. Aboutalebi et al [35] proposed a multi-scale encoder-decoder self-attention (MEDUSA) model to solve the problem of overlapping image appearance. The model improved the ability to model global remote spatial context by introducing self-attention modules, achieving good classification performance on several datasets.…”