Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers

Abstract

Most polyp segmentation methods use CNNs as their backbone, leading to two key issues when exchanging information between the encoder and decoder: 1) taking into account the differences in contribution between different-level features; and 2) designing effective mechanism for fusing these features. Different from existing CNN-based methods, we adopt a transformer encoder, which learns more powerful and robust representations. In addition, considering the image acquisition influence and elusive properties of polyps, we introduce three novel modules, including a cascaded fusion module (CFM), a camouflage identification module (CIM), a and similarity aggregation module (SAM). Among these, the CFM is used to collect the semantic and location information of polyps from high-level features, while the CIM is applied to capture polyp information disguised in low-level features. With the help of the SAM, we extend the pixel features of the polyp area with high-level semantic position information to the entire polyp area, thereby effectively fusing cross-level features. The proposed model, named Polyp-PVT, effectively suppresses noises in the features and significantly improves their expressive capabilities. Extensive experiments on five widely adopted datasets show that the proposed model is more robust to various challenging situations (e.g., appearance changes, small objects) than existing methods, and achieves the new state-of-the-art performance. The proposed model is available at https://github.com/DengPingFan/Polyp-PVT.

PVT v2: Improved baselines with Pyramid Vision Transformer

“…For example, Vision Transformer (ViT) [1] first showed that a pure transformer can archive stateof-the-art performance in image classification. The Pyramid Vision Transformer (PVT v1) [3] showed that a pure transformer backbone can also surpass CNN counterparts for dense prediction tasks such as detection and segmentation [9][10][11]. Later, Swin transformer [5], CoaT [6], LeViT [7], and Twins [8] further improved classification, detection, and segmentation performance with transformer backbones.…”

Automatic Detection and Segmentation of Thrombi in Abdominal Aortic Aneurysms Using a Mask Region-Based Convolutional Neural Network with Optimized Loss Functions

“…Nanni et al [ 36 ] proposed encoder–decoder ensemble classifiers that can be used for semantic segmentation and introduced a novel loss function that results from the combination of Dice loss and a structural similarity index (SSIM). Dong et al [ 37 ] presented a pyramid vision transformer backbone as an encoder for the extraction of robust features that has three tight components: a cascaded fusion module (CFM), camouflage identification module (CIM), and similarity aggregation module (SAM). The sum of the IoU and weighted binary cross-entropy loss is used as the loss function.…”

A Standardized Approach for Skin Detection: Analysis of the Literature and Case-Studies

“…• In [1] and [60] several deep learning segmentation approaches are compared, SegNet, U-Net, DeepLabv3+, HarD-NetMSEG (Harmonic Densely Connected Network) 1 [61] and Polyp-PVT [62] a deep learning segmentation model based on a transformer encoder, i.e. PVT (Pyramid Vision Transformer) 2 .…”

“…• number of epoch=10 (using the simple data augmentation approach DA1, see section 3.3) or 15 (the latter more complex data augmentation approach DA2, see section 3. We present an ensemble based on DeepLabV3+, HarDNet-MSEG [61], Polyp-PVT [62], and Hybrid Semantic Network (HSN) [79]. HarD-Net-MSEG (Harmonic Densely Connected Network) [61] is a model influenced by densely connected networks, that can reduce memory consumption by diminishing aggregation with the reduction of most connection layers to the DenseNet layer.…”