Performance and Limitation of Machine Learning Algorithms for Diabetic Retinopathy Screening: Meta-analysis

Abstract

Background Diabetic retinopathy (DR), whose standard diagnosis is performed by human experts, has high prevalence and requires a more efficient screening method. Although machine learning (ML)–based automated DR diagnosis has gained attention due to recent approval of IDx-DR, performance of this tool has not been examined systematically, and the best ML technique for use in a real-world setting has not been discussed. Objective The aim of this study was to systematically examine the overall diagnostic accuracy of ML in diagnosing DR of different categories based on color fundus photographs and to determine the state-of-the-art ML approach. Methods Published studies in PubMed and EMBASE were searched from inception to June 2020. Studies were screened for relevant outcomes, publication types, and data sufficiency, and a total of 60 out of 2128 (2.82%) studies were retrieved after study selection. Extraction of data was performed by 2 authors according to PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses), and the quality assessment was performed according to the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2). Meta-analysis of diagnostic accuracy was pooled using a bivariate random effects model. The main outcomes included diagnostic accuracy, sensitivity, and specificity of ML in diagnosing DR based on color fundus photographs, as well as the performances of different major types of ML algorithms. Results The primary meta-analysis included 60 color fundus photograph studies (445,175 interpretations). Overall, ML demonstrated high accuracy in diagnosing DR of various categories, with a pooled area under the receiver operating characteristic (AUROC) ranging from 0.97 (95% CI 0.96-0.99) to 0.99 (95% CI 0.98-1.00). The performance of ML in detecting more-than-mild DR was robust (sensitivity 0.95; AUROC 0.97), and by subgroup analyses, we observed that robust performance of ML was not limited to benchmark data sets (sensitivity 0.92; AUROC 0.96) but could be generalized to images collected in clinical practice (sensitivity 0.97; AUROC 0.97). Neural network was the most widely used method, and the subgroup analysis revealed a pooled AUROC of 0.98 (95% CI 0.96-0.99) for studies that used neural networks to diagnose more-than-mild DR. Conclusions This meta-analysis demonstrated high diagnostic accuracy of ML algorithms in detecting DR on color fundus photographs, suggesting that state-of-the-art, ML-based DR screening algorithms are likely ready for clinical applications. However, a significant portion of the earlier published studies had methodology flaws, such as the lack of external validation and presence of spectrum bias. The results of these studies should be interpreted with caution.

Infodemic Management Using Digital Information and Knowledge Cocreation to Address COVID-19 Vaccine Hesitancy: Case Study From Ghana

“…Talkwalker cannot carry out such an interpretation, which aligns with a number of studies that have pointed out the limitations of machines. Although machine learning methods have been developed to solve real-world problems, they are not sufficient by themselves in critical decision-making approaches [ 38 , 39 ]. Digital platforms still have limitations to interpret and contextualize data [ 40 ].…”

Opportunities of Digital Infrastructures for Disease Management—Exemplified on COVID-19-Related Change in Diagnosis Counts for Diabetes-Related Eye Diseases

“…From a medical perspective, the availability of structured data builds the base for additional tools, such as machine learning algorithms (e.g., [ 23 ]) and decision support systems (e.g., [ 24 , 25 ]), which result in suggestions for diagnostic and therapy options, especially, but not limited to, diabetes-related eye-diseases.…”

Validation of an Automated Screening System for Diabetic Retinopathy Operating under Real Clinical Conditions

“…This suggests that the mechanisms underlying decision-making are different in algorithms and clinicians. The grading criteria of the automatic systems may have been more demanding than those of the humans, but what is evident is that their implementation is invariable and not affected by the inherent inconsistency of human subjectivity [ 29 ].…”

“…IDx-DR v2 achieved 100% sensitivity and 81.82% specificity for derivable DR and 100% sensitivity and 94.64% specificity for STDR in the sample examined [ 38 ]. A meta-analysis demonstrated that ML algorithms have a high diagnostic accuracy for the diagnosis of DR on color fundus photographs suggesting that they may also be ready for clinical application in screening programs [ 29 ]. However, early results published in relation to this ML had methodological inconsistencies, such as lack of external validation and the presence of biases.…”