TGANet: Text-Guided Attention for Improved Polyp Segmentation

Abstract

Colonoscopy is a gold standard procedure but is highly operatordependent. Automated polyp segmentation, a precancerous precursor, can minimize missed rates and timely treatment of colon cancer at an early stage. Even though there are deep learning methods developed for this task, variability in polyp size can impact model training, thereby limiting it to the size attribute of the majority of samples in the training dataset that may provide sub-optimal results to differently sized polyps. In this work, we exploit size-related and polyp number-related features in the form of text attention during training. We introduce an auxiliary classification task to weight the text-based embedding that allows network to learn additional feature representations that can distinctly adapt to differently sized polyps and can adapt to cases with multiple polyps. Our experimental results demonstrate that these added text embeddings improve the overall performance of the model compared to state-of-the-art segmentation methods. We explore four different datasets and provide insights for size-specific improvements. Our proposed text-guided attention network (TGANet) can generalize well to variable-sized polyps in different datasets.

CADNet: an advanced architecture for automatic detection of coronary artery calcification and shadow border in intravascular ultrasound (IVUS) images

“…Due to the scarcity of a limited amount of annotated data in the case of biomedical imaging, data augmentation is preferred [ 61 ]. Various augmentation (operation) types have been investigated in the literature to create fresh training images: When analysing visual documents, elastic distortions are used [ 62 ]; other examples include scaling, translation, shearing, flipping, and rotation transformations [ 19 , 20 , 27 , 39 , 51 , 63 – 66 ]. However, not every augmentation operation will be useful in medical settings.…”

“…Every pixel in the image is classified using semantic segmentation into one of the specified classes [ 26 ]. In the medical field, deep learning-based methods are frequently used in the diagnosis of breast tumors [ 27 ], Covid-19 lung infection [ 28 ], coronary segmentation [ 19 – 21 , 29 – 35 ], Alzheimer disease prediction [ 36 ], skin lesion segmentation [ 37 ], dermatological diseases [ 38 ] and polyp segmentation [ 39 ] to mention a few. Table 1 depicts various areas of medical image segmentation where deep learning is widely used.…”

Where do we stand in AI for endoscopic image analysis? Deciphering gaps and future directions

“…For segmentation, current developments are based widely on encoder-decoder architectures [56][57][58] . Tomar et al 57 proposed to combine text label embedding as an attention mechanism for effective polyp segmentation and to improve generalisability.…”

“…For segmentation, current developments are based widely on encoder-decoder architectures [56][57][58] . Tomar et al 57 proposed to combine text label embedding as an attention mechanism for effective polyp segmentation and to improve generalisability. During training auxiliary classification task for learning size-related and polyp number-related features was trained and embedded with the segmentation network alongside showing improvement of up to 2% over SOTA methods on four public datasets.…”

RUPNet: residual upsampling network for real-time polyp segmentation

“…Recently, Biffi et al 12 proposed an intelligent medical device for real-time optical characterization of colonoscopy polyps which can be also integrated easily into clinics. Besides that, there are several recent works on colonoscopy targeting polyps attributes such as size and count, 13 flat, sessile or diminutive polyps, 14 out-of-distribution polyp detection through cross dataset test, 14,15 and synthetic data generation for improving the performance of the network. 16 As the study of colonoscopy cancer is becoming mature more emphasis is being given towards exploring the possibilities of new technological advancement such as diagnostic classification, risk stratification, and outcome examination.…”