Self-Supervised Feature Learning via Exploiting Multi-Modal Data for Retinal Disease Diagnosis

Abstract

The automatic diagnosis of various retinal diseases from fundus images is important to support clinical decisionmaking. However, developing such automatic solutions is challenging due to the requirement of a large amount of humanannotated data. Recently, unsupervised/self-supervised feature learning techniques receive a lot of attention, as they do not need massive annotations. Most of the current self-supervised methods are analyzed with single imaging modality and there is no method currently utilize multi-modal images for better results. Considering that the diagnostics of various vitreoretinal diseases can greatly benefit from another imaging modality, e.g., FFA, this paper presents a novel self-supervised feature learning method by effectively exploiting multi-modal data for retinal disease diagnosis. To achieve this, we first synthesize the corresponding FFA modality and then formulate a patient feature-based softmax embedding objective. Our objective learns both modality-invariant features and patient-similarity features. Through this mechanism, the neural network captures the semantically shared information across different modalities and the apparent visual similarity between patients. We evaluate our method on two public benchmark datasets for retinal disease diagnosis. The experimental results demonstrate that our method clearly outperforms other self-supervised feature learning methods and is comparable to the supervised baseline. Our code is available at GitHub 1. Index Terms-Retinal disease diagnosis, self-supervised learning, multi-modal data I. INTRODUCTION C OLOR fundus photography has been widely used in clinical practice to evaluate various conventional ophthalmic diseases, e.g., age-related macular degeneration (AMD) [1], pathologic myopia (PM) [2], and diabetic retinopathy [3, 4]. Recently, deep learning has shown very good performance on a variety of automatic ophthalmic disease detection problems from fundus images [5-7], and these techniques can help ophthalmologists in decision making. The success is attributed to the learned representative features from fundus images, which requires a large amount of training data with massive human annotations. However, it is tedious and expensive to annotate the fundus images, since experts are needed to provide reliable labels. Hence, in this paper, our goal is to learn the representative features from data itself, without any human

Self-supervised multimodal reconstruction pre-training for retinal computer-aided diagnosis

“…Second, under this same AMD and glaucoma use cases, we compare the proposed selfsupervised pre-training method with commonly used fully-supervised baseline approaches, based on initializing the diagnosis network with random weights, and using ImageNet classification pre-training. Finally, our work is compared with (Li et al, 2020) which, to the best of our knowledge, is the only related work in the literature using selfsupervised approaches for retinal image analysis. Specifically.…”

“…Specifically. In order to provide comparable results, we follow the exact same experimental setting used in Li et al (2020), to provide AMD and pathological myopia (PM) diagnosis from retinographies. In this case, the proposed multimodal self-supervised pre-training framework is directly applied without bells and whistles.…”

“…More recently, Taleb et al (2020) extended several state-of-the-art self-supervised approaches to 3D medical data, including, e.g., jigsaw puzzles or instance discrimination with contrastive learning. Similarly, the instance discrimination paradigm was extended by including the synthesis of a complementary image modality as an additional transformation of the input image (Li, Jia, Islam, Yu, & Xing, 2020). Additionally, other novel self-supervised paradigms have been directly proposed in the medical imaging field.…”

“…However, self-supervision based on multimodal reconstruction of medical images has not yet been explored for improving deep learning CAD systems. In this regard, although Li et al (2020) aim at using complementary modalities to aid self-supervision, their setting is not based on direct prediction. Instead, they explore the use of synthetic complementary image modalities as an additional augmentation strategy in a contrastive learning instance discrimination setting.…”

Integrating Enhanced Sparse Autoencoder-Based Artificial Neural Network Technique and Softmax Regression for Medical Diagnosis

“…Imbalanced data are a classification problem in which the number of instances per class is not uniformly distributed. Recently, unsupervised feature learning methods have received massive attention since they do not entirely rely on labeled data [5], and are suitable for training models when the data are imbalanced.…”