Conv-ViT: A Convolution and Vision Transformer-Based Hybrid Feature Extraction Method for Retinal Disease Detection

Abstract

The current advancement towards retinal disease detection mainly focused on distinct feature extraction using either a convolutional neural network (CNN) or a transformer-based end-to-end deep learning (DL) model. The individual end-to-end DL models are capable of only processing texture or shape-based information for performing detection tasks. However, extraction of only texture- or shape-based features does not provide the model robustness needed to classify different types of retinal diseases. Therefore, concerning these two features, this paper developed a fusion model called ‘Conv-ViT’ to detect retinal diseases from foveal cut optical coherence tomography (OCT) images. The transfer learning-based CNN models, such as Inception-V3 and ResNet-50, are utilized to process texture information by calculating the correlation of the nearby pixel. Additionally, the vision transformer model is fused to process shape-based features by determining the correlation between long-distance pixels. The hybridization of these three models results in shape-based texture feature learning during the classification of retinal diseases into its four classes, including choroidal neovascularization (CNV), diabetic macular edema (DME), DRUSEN, and NORMAL. The weighted average classification accuracy, precision, recall, and F1 score of the model are found to be approximately 94%. The results indicate that the fusion of both texture and shape features assisted the proposed Conv-ViT model to outperform the state-of-the-art retinal disease classification models.

Stitched vision transformer for age-related macular degeneration detection using retinal optical coherence tomography images

“…However, it leads to an increase in the dimensionality of the feature vectors. Furthermore, it does not address the issue of high data requirements, a characteristic commonly associated with purely attention-based models such as ViTs [ 65 ]. Table 2 briefly reviews the studies that utilized hybrid and ensemble methods in 2023.…”

“…In numerous studies in the realm of medical image classification, different CNNs have served as classifiers or feature extractors because of their advantage in automatic generalizable feature extraction of OCT images, surpassing traditional image processing-based feature extractors. Some of the reviewed studies have incorporated attention blocks into their proposed hybrid architectures as global feature selectors to improve the performance of traditional CNNs [ 56 , 65 ]. Global attention blocks [ 56 ] generate attention maps by utilizing multiple low-cost max and average pooling layers, and do not include the standard high performance self-attention.…”

“…Global attention blocks [ 56 ] generate attention maps by utilizing multiple low-cost max and average pooling layers, and do not include the standard high performance self-attention. The CNN-transformer [ 65 ] architecture is an ensemble of the ViT, ResNet50, and Inception-v3 models and does not address the high-data requirement of ViT, particularly in non-transfer learning scenarios. MedViT is the backbone model of the proposed stitching method in this study.…”

Laryngeal Cancer Detection and Classification Using Aquila Optimization Algorithm With Deep Learning on Throat Region Images

“…Primarily, the Inceptionv3 model is used for the generation of feature vectors. The InceptionV3 is selected in the initial feature extraction model, due to its capability to extract higherlevel features with different variations of the 277 filters [23]. Furthermore, without compromising the model efficacy, the architecture of InceptionV3 presents the features of reduction dimension by applying two (3x3) layers rather than a single (5x5) convolutional layer because with a similar amount of filters 𝑎(5x5) convolutional layer is 25/9 = 2.78 times computational expensive than 𝑎(3x3) convolution layers.…”

Comparison of Different Methods for Building Ensembles of Convolutional Neural Networks

“…Artificial neural networks (ANNs), which were initially developed in the 1950s, have had a checkered history, at times appreciated for their unique computational capabilities and at other times disparaged for being no better than statistical methods. Opinions shifted about a decade ago with deep neural networks, whose performance swiftly overshadowed that of other learners across various scientific (e.g., [1,2]), medical (e.g., [3,4]), and engineering domains (e.g., [5,6]). The prowess of deep learners is especially exemplified by the remarkable achievements of Convolutional Neural Networks (CNNs), one of the most renowned and robust deep-learning architectures.…”