Multimodal, multitask, multiattention (M3) deep learning detection of reticular pseudodrusen: Toward automated and accessible classification of age-related macular degeneration

Abstract

Objective Reticular pseudodrusen (RPD), a key feature of age-related macular degeneration (AMD), are poorly detected by human experts on standard color fundus photography (CFP) and typically require advanced imaging modalities such as fundus autofluorescence (FAF). The objective was to develop and evaluate the performance of a novel multimodal, multitask, multiattention (M3) deep learning framework on RPD detection. Materials and Methods A deep learning framework (M3) was developed to detect RPD presence accurately using CFP alone, FAF alone, or both, employing >8000 CFP-FAF image pairs obtained prospectively (Age-Related Eye Disease Study 2). The M3 framework includes multimodal (detection from single or multiple image modalities), multitask (training different tasks simultaneously to improve generalizability), and multiattention (improving ensembled feature representation) operation. Performance on RPD detection was compared with state-of-the-art deep learning models and 13 ophthalmologists; performance on detection of 2 other AMD features (geographic atrophy and pigmentary abnormalities) was also evaluated. Results For RPD detection, M3 achieved an area under the receiver-operating characteristic curve (AUROC) of 0.832, 0.931, and 0.933 for CFP alone, FAF alone, and both, respectively. M3 performance on CFP was very substantially superior to human retinal specialists (median F1 score = 0.644 vs 0.350). External validation (the Rotterdam Study) demonstrated high accuracy on CFP alone (AUROC, 0.965). The M3 framework also accurately detected geographic atrophy and pigmentary abnormalities (AUROC, 0.909 and 0.912, respectively), demonstrating its generalizability. Conclusions This study demonstrates the successful development, robust evaluation, and external validation of a novel deep learning framework that enables accessible, accurate, and automated AMD diagnosis and prognosis.

Classification of dry and wet macular degeneration based on the ConvNeXT model

“… Priya and Aruna (2014) used a machine learning approach to extract retinal image features for classification, with the support vector machine (SVM) classifier achieving a maximum accuracy of 96%. Chen et al (2021) used a multimodal deep-learning framework to automatically classify macular degeneration with a maximum accuracy of 90.65%. Traditional machine learning algorithms exhibit poor generalization performance and are prone to over-fitting problems.…”

“…Few studies have classified dry and wet macular degeneration. Priya and Aruna (2014) used machine learning methods, such as probabilistic neural networks, to extract retinal features and classify dry and wet macular degeneration with a maximum accuracy, sensitivity, and specificity of 96, 96.96, and 94.11%, respectively, while Chen et al (2021) used a multimodal deep learning framework to automate the detection of dry and wet macular degeneration with a maximum accuracy, sensitivity, and specificity of 90.65, 68.92, and 98.53%, respectively. In some of these studies, the evaluation indicators of the models were not sufficiently comprehensive, the sensitivity and specificity values of some models varied widely, and some models studied only dichotomous categories, with limited practical application.…”

Artificial intelligence in chorioretinal pathology through fundoscopy: a comprehensive review

“…CNNs like ResNet, Inception-ResNet-V2, and DeepSeeNet have achieved great accuracies in AMD classification, with AUCs above 0.970 [ 47 – 50 ]. Innovative techniques such as UWF-based CNN and multimodal frameworks improve AMD detection, resulting in impressive AUCs [ 51 , 52 ].…”

Detection of reticular pseudodrusen on optical coherence tomography images

“…Their proposed network slightly enhanced the performance obtained using FAF images only with AUC of 0.933. 7 To our knowledge, only Schwartz et al 8 have attempted to detect both RPD and conventional drusen from OCT scans, where their model achieved AUC of 0.99. However, they detected the presence of RPD or drusen without distinguishing between them.…”

“…Their FAF model outperformed the CFP model and surpassed the performance of ophthalmologists, 6 where it achieved area under the receiver-operating characteristic curve (AUC) of 0.939. Chen et al 7 developed a multi-modal network that uses both CFP and FAF in RPD detection. Their proposed network slightly enhanced the performance obtained using FAF images only with AUC of 0.933.…”