Transformers Improve Breast Cancer Diagnosis from Unregistered Multi-View Mammograms

Abstract

Deep convolutional neural networks (CNNs) have been widely used in various medical imaging tasks. However, due to the intrinsic locality of convolution operations, CNNs generally cannot model long-range dependencies well, which are important for accurately identifying or mapping corresponding breast lesion features computed from unregistered multiple mammograms. This motivated us to leverage the architecture of Multi-view Vision Transformers to capture long-range relationships of multiple mammograms from the same patient in one examination. For this purpose, we employed local transformer blocks to separately learn patch relationships within four mammograms acquired from two-view (CC/MLO) of two-side (right/left) breasts. The outputs from different views and sides were concatenated and fed into global transformer blocks, to jointly learn patch relationships between four images representing two different views of the left and right breasts. To evaluate the proposed model, we retrospectively assembled a dataset involving 949 sets of mammograms, which included 470 malignant cases and 479 normal or benign cases. We trained and evaluated the model using a five-fold cross-validation method. Without any arduous preprocessing steps (e.g., optimal window cropping, chest wall or pectoral muscle removal, two-view image registration, etc.), our four-image (two-view-two-side) transformer-based model achieves case classification performance with an area under ROC curve (AUC = 0.818 ± 0.039), which significantly outperforms AUC = 0.784 ± 0.016 achieved by the state-of-the-art multi-view CNNs (p = 0.009). It also outperforms two one-view-two-side models that achieve AUC of 0.724 ± 0.013 (CC view) and 0.769 ± 0.036 (MLO view), respectively. The study demonstrates the potential of using transformers to develop high-performing computer-aided diagnosis schemes that combine four mammograms.

SaTransformer: Semantic‐aware transformer for breast cancer classification and segmentation

“…Their approach combines a residual convolutional network with a transformer encoder that incorporates multiple layer perceptron (MLP) modules. Chen et al [30] introduced multi-view vision transformers (MVTs) for diagnosing breast cancer from unregistered multi-view mammograms. The MVTs used local and global transformer blocks to capture within-mammogram and inter-mammogram dependencies, and it was capable of processing four-view mammograms simultaneously.…”

“…Chen et al. [30] introduced multi‐view vision transformers (MVTs) for diagnosing breast cancer from unregistered multi‐view mammograms. The MVTs used local and global transformer blocks to capture within‐mammogram and inter‐mammogram dependencies, and it was capable of processing four‐view mammograms simultaneously.…”

ETECADx: Ensemble Self-Attention Transformer Encoder for Breast Cancer Diagnosis Using Full-Field Digital X-ray Breast Images

“…The suggested model outperformed CNN models of DenseNet201, ResNet101, and VGG19 by achieving 96.29% precision, a 96.15% F1-score, and 95.29% accuracy. Chen et al [42] suggested using the local and global transformer blocks to model within four mammograms taken from both views for each side. The four images were then combined into a single sequence global transformer and passed into the MLP head for classification.…”

Convolutional Networks and Transformers for Mammography Classification: An Experimental Study

“…Currently, the literature using transformer-based networks for mammography analysis is still limited, especially in the case of whole image classification. Very recently, Chen et al [ 36 ] proposed a transformer-based method to classify multi-view mammograms, that achieve an AUC of on a dataset consisting of 3796 images, surpassing the state-of-the-art multi-view CNN model. Simultaneously with our research, Swin transformer has been tested in single-view mammography classification obtaining an AUC of on DDSM dataset [ 37 ].…”