PCAT-UNet: UNet-like network fused convolution and transformer for retinal vessel segmentation

Abstract

The accurate segmentation of retinal vessels images can not only be used to evaluate and monitor various ophthalmic diseases, but also timely reflect systemic diseases such as diabetes and blood diseases. Therefore, the study on segmentation of retinal vessels images is of great significance for the diagnosis of visually threatening diseases. In recent years, especially the convolutional neural networks (CNN) based on UNet and its variant have been widely used in various medical image tasks. However, although CNN has achieved excellent performance, it cannot learn global and long-distance semantic information interaction well due to the local computing characteristics of convolution operation, which limits the development of medical image segmentation tasks. Transformer, currently popular in computer vision, has global computing features, but due to the lack of low-level details, local feature information extraction is insufficient. In this paper, we propose Patches Convolution Attention based Transformer UNet (PCAT-UNet), which is a U-shaped network based on Transformer with a Convolution branch. We use skip connection to fuse the deep and shallow features of both sides. By taking advantage of the complementary advantages of both sides, we can effectively capture the global dependence relationship and the details of the underlying feature space, thus improving the current problems of insufficient extraction of retinal micro vessels feature information and low sensitivity caused by easily predicting of pixels as background. In addition, our method enables end-to-end training and rapid inference. Finally, three publicly available retinal vessels datasets (DRIVE, STARE and CHASE_DB1) were used to evaluate PCAT-UNet. The experimental results show that the proposed PCAT-UNET method achieves good retinal vessel segmentation performance on these three datasets, and is superior to other architectures in terms of AUC, Accuracy and Sensitivity performance indicators. AUC reached 0.9872, 0.9953 and 0.9925, Accuracy reached 0.9622, 0.9796 and 0.9812, Sensitivity reached 0.8576, 0.8703 and 0.8493, respectively. In addition, PCAT-UNET also achieved good results in two other F1-Score and Specificity indicators.

TiM-Net: Transformer in M-Net for Retinal Vessel Segmentation

“…Cao [ 48 ] proposed a pure Transformer network to classify and segment images with great success. Chen et al [ 26 ] proposed the PCAT-UNet model that absorbs a modified Transformer module into U-Net. However, due to the lack of the ability to capture the long-range relationship, noisy features are obtained after multiple convolutions, which affects the final performance.…”

“…Reviewing the work of [ 17 , 18 , 24 – 26 ], we found the following problems: (1) most studies use complex structures, which may lower their practicalities; (2) traditional encoders cannot model long-range relationships and are prone to noisy interference; and (3) the side output layer only uses a single-layer output, which cannot make full use of the complementarity of different layers. The complementarity helps recover the feature maps well.…”

“…Recently, owing to the great success of Transformer in the CV field, TransUNet [ 24 ] and TransFuse [ 25 ] have been proposed by combining Transformer and U-Net. Similarly, Chen proposed the patches convolution attention-based Transformer U-Net (PCAT-UNet) [ 26 ] model that inserts a modified Transformer module into U-Net. Although better performance can be observed, these Transformer-based models are complex and time-consuming, which will affect their practicalities to some degree.…”

Recent Progress in Transformer-based Medical Image Analysis

“…For mainstream classification and segmentation, we further divide them Figure 6: Examples of all the modalities included in this work. Sequences are X-ray [28], US [36], MRI [37], CT [38], WCE [39], OCT [40], fundus camera [41], camera [42], scanner [43], microscope [44], dermoscopic [45], colonoscopy [46], and laryngoscopy [47]. Images are preprocessed to greyscale to prevent readers from being uncomfortable.…”

“…Fundus camera. Chen et al [41] designed a model called PCAT-UNet to classify fundus vessel images [157,158,159]. The proposed U-shape network contains two main components named Figure 8: Segmentation results using different segmentation methods on the US dataset [156].…”