Validating automated eye disease screening AI algorithm in community and in-hospital scenarios

Abstract

Purpose:To assess the accuracy and robustness of the AI algorithm for detecting referable diabetic retinopathy (RDR), referable macular diseases (RMD), and glaucoma suspect (GCS) from fundus images in community and in-hospital screening scenarios.MethodsWe collected two color fundus image datasets, namely, PUMCH (556 images, 166 subjects, and four camera models) and NSDE (534 images, 134 subjects, and two camera models). The AI algorithm generates the screening report after taking fundus images. The images were labeled as RDR, RMD, GCS, or none of the three by 3 licensed ophthalmologists. The resulting labels were treated as “ground truth” and then were used to compare against the AI screening reports to validate the sensitivity, specificity, and area under the receiver operating characteristic curve (AUC) of the AI algorithm.ResultsOn the PUMCH dataset, regarding the prediction of RDR, the AI algorithm achieved overall results of 0.950 ± 0.058, 0.963 ± 0.024, and 0.954 ± 0.049 on sensitivity, specificity, and AUC, respectively. For RMD, the overall results are 0.919 ± 0.073, 0.929 ± 0.039, and 0.974 ± 0.009. For GCS, the overall results are 0.950 ± 0.059, 0.946 ± 0.016, and 0.976 ± 0.025.ConclusionThe AI algorithm can work robustly with various fundus camera models and achieve high accuracies for detecting RDR, RMD, and GCS.

Artificial intelligence promotes the diagnosis and screening of diabetic retinopathy

“…In China, training and validation data for AI algorithms are vast due to a rather centralized healthcare system and the largest population of DR ( 93 ). Hundreds of new start-up companies working on AI applications to healthcare in China have emerged to improve business, and several DR AI-based screening tools have acquired the certificate of medical device Class III approved by NMPA as pioneers, e.g., Silicon Intelligence, Airdoc, Vistel, and especially Intelligent Healthcare of Baidu that developed the first granted algorithm working robustly with various fundus camera models and achieving high accuracies for detecting multiple ophthalmic diseases ( 94 , 95 ). Undoubtedly, the real-world deployment of these new systems in multiple settings will be full of challenges not only in AI diagnostic technologies but also in the marketing pattern and policy-making.…”

A 178-clinical-center experiment of integrating AI solutions for lung pathology diagnosis

“…One of the problems of such studies is that algorithms are often tested on the internal datasets sampled from the same source as training images, which makes the results subjected to dataset bias, potential data contamination, and low data representativeness. To mitigate these issues, large-scale multicenter studies have been conducted allowing the researchers to better estimate the prospects of AI for, for example, eye disease diagnosis 11 , 12 . Such studies are, however, expensive and lengthy.…”

Corneal Sub-Basal Nerve Plexus in Non-Diabetic Small Fiber Polyneuropathies and the Diagnostic Role of In Vivo Corneal Confocal Microscopy

“…Finally, the rapidly developing use of artificial intelligence in ophthalmology now permits the analysis of data with automated software in the context of screening ocular diseases and detecting early alterations with high sensitivity and specificity. The machine learning used in IVCM data analysis could provide precise information by using algorithms for fully automated nerve assessments to detect small fiber alterations [ 103 , 104 ].…”