Evaluating transformer-based semantic segmentation networks for pathological image segmentation

Nguyen, Cam; Asad, Zuhayr; Deng, Ruining; Huo, Yuankai

doi:10.1117/12.2611177

Cited by 19 publications

(14 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similarly in medical imaging, recent years have seen emergence of many vision transformers models replacing CNNs. Such models have demonstrated enhanced global context, leading to both qualitative and quantitative improvements for histopathology [57] and brain tumor segmentation [58], [59], tumor classification [60] and polyp detection [61]. Pure (endto-end) transformers have even shown advantages over CNNs in the low-data regime for brain segmentation [62].…”

Section: B Attentionmentioning

confidence: 99%

Surface Vision Transformers: Flexible Attention-Based Modelling of Biomedical Surfaces

Dahan¹,

Xu²,

Williams³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent state-of-the-art performances of Vision Transformers (ViT) in computer vision tasks demonstrate that a general-purpose architecture, which implements long-range self-attention, could replace the local feature learning operations of convolutional neural networks. In this paper, we extend ViTs to surfaces by reformulating the task of surface learning as a sequence-tosequence learning problem, by proposing patching mechanisms for general surface meshes. Sequences of patches are then processed by a transformer encoder and used for classification or regression. We validate our method on a range of different biomedical surface domains and tasks: brain age prediction in the developing Human Connectome Project (dHCP), fluid intelligence prediction in the Human Connectome Project (HCP), and coronary artery calcium score classification using surfaces from the Scottish Computed Tomography of the Heart (SCOT-HEART) dataset, and investigate the impact of pretraining and data augmentation on model performance. Results suggest that Surface Vision Transformers (SiT) demonstrate consistent improvement over geometric deep learning methods for brain age and fluid intelligence prediction and achieve comparable performance on calcium score classification to standard metrics used in clinical practice. Furthermore, analysis of transformer attention maps offers clear and individualised predictions of the features driving each task. Code is available on Github: https://github.com/ metrics-lab/surface-vision-transformers.

show abstract

Section: B Attentionmentioning

confidence: 99%

Surface Vision Transformers: Flexible Attention-Based Modelling of Biomedical Surfaces

Dahan¹,

Xu²,

Williams³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The authors tested their framework in different computational pathology problems. Using high-resolution images, Nguyen et al [130] performed a comparative analysis using architectures based on CNN and Transformer modules and reported improved results with the Transformer-based approaches. In a similar approach, using hyperspectral image data, Yun et al [131] proposed an encoder-decoder architecture with CNN and Transformer blocks to extract and model spatial and spectral features, achieving competitive results against other methodologies.…”

Section: ) Breast Lesion Segmentationmentioning

confidence: 99%

A Survey on Attention Mechanisms for Medical Applications: are we Moving Toward Better Algorithms?

et al. 2022

View full text Add to dashboard Cite

The increasing popularity of attention mechanisms in deep learning algorithms for computer vision and natural language processing made these models attractive to other research domains. In healthcare, there is a strong need for tools that may improve the routines of the clinicians and the patients. Naturally, the use of attention-based algorithms for medical applications occurred smoothly. However, being healthcare a domain that depends on high-stake decisions, the scientific community must ponder if these high-performing algorithms fit the needs of medical applications. With this motto, this paper extensively reviews the use of attention mechanisms in machine learning methods (including Transformers) for several medical applications based on the types of tasks that may integrate several works pipelines of the medical domain. This work distinguishes itself from its predecessors by proposing a critical analysis of the claims and potentialities of attention mechanisms presented in the literature through an experimental case study on medical image classification with three different use cases. These experiments focus on the integrating process of attention mechanisms into established deep learning architectures, the analysis of their predictive power, and a visual assessment of their saliency maps generated by post-hoc explanation methods. This paper concludes with a critical analysis of the claims and potentialities presented in the literature about attention mechanisms and proposes future research lines in medical applications that may benefit from these frameworks.

show abstract

“…Transformers originated from and are most publicized in the NLP domain (e.g., BERT [29], GPT-3 [30], T5 [31]), including notable examples in the medical domain [32][33][34][35], and have begun to demonstrate state-of-the-art performance in general imaging applications [36][37][38] including computational pathology [39][40][41]. While the original transformer used an encoder-decoder architecture due to its primary task of machine translation [28], many modern transformers for classification tasks (including ours) use an encoder-only architecture.…”

Section: Introductionmentioning

confidence: 99%

“…The past two years have seen a surge in popularity of transformer modeling for common computational pathology tasks such as WSI segmentation [40,43,44] and histology image classification [41,[45][46][47]. Transformers have also been used for pathologist-level question-answering from histological imaging [39], predicting pathologists' visual attention [48], and for pathology text mining [49].…”

Section: Introductionmentioning

confidence: 99%

Spatially Aware Transformer Networks for Contextual Prediction of Diabetic Nephropathy Progression from Whole Slide Images

Shickel

Lucarelli

Rao

et al. 2023

Preprint

View full text Add to dashboard Cite

Diabetic nephropathy (DN) in the context of type 2 diabetes is the leading cause of end-stage renal disease (ESRD) in the United States. DN is graded based on glomerular morphology and has a spatially heterogeneous presentation in kidney biopsies that complicates pathologists predictions of disease progression. Artificial intelligence and deep learning methods for pathology have shown promise for quantitative pathological evaluation and clinical trajectory estimation; but, they often fail to capture large-scale spatial anatomy and relationships found in whole slide images (WSIs). In this study, we present a transformer-based, multi-stage ESRD prediction framework built upon nonlinear dimensionality reduction, relative Euclidean pixel distance embeddings between every pair of observable glomeruli, and a corresponding spatial self-attention mechanism for a robust contextual representation. We developed a deep transformer network for encoding WSI and predicting future ESRD using a dataset of 56 kidney biopsy WSIs from DN patients at Seoul National University Hospital. Using a leave-one-out cross-validation scheme, our modified transformer framework outperformed RNNs, XGBoost, and logistic regression baseline models, and resulted in an area under the receiver operating characteristic curve (AUC) of 0.97 (95% CI: 0.90-1.00) for predicting two-year ESRD, compared with an AUC of 0.86 (95% CI: 0.66-0.99) without our relative distance embedding, and an AUC of 0.76 (95% CI: 0.59-0.92) without a denoising autoencoder module. While the variability and generalizability induced by smaller sample sizes are challenging, our distance-based embedding approach and overfitting mitigation techniques yielded results that suggest opportunities for future spatially aware WSI research using limited pathology datasets.

show abstract

Evaluating transformer-based semantic segmentation networks for pathological image segmentation

Cited by 19 publications

References 4 publications

Surface Vision Transformers: Flexible Attention-Based Modelling of Biomedical Surfaces

Surface Vision Transformers: Flexible Attention-Based Modelling of Biomedical Surfaces

A Survey on Attention Mechanisms for Medical Applications: are we Moving Toward Better Algorithms?

Spatially Aware Transformer Networks for Contextual Prediction of Diabetic Nephropathy Progression from Whole Slide Images

Contact Info

Product

Resources

About