Validating automated eye disease screening AI algorithm in community and in-hospital scenarios

Abstract

Purpose:To assess the accuracy and robustness of the AI algorithm for detecting referable diabetic retinopathy (RDR), referable macular diseases (RMD), and glaucoma suspect (GCS) from fundus images in community and in-hospital screening scenarios.MethodsWe collected two color fundus image datasets, namely, PUMCH (556 images, 166 subjects, and four camera models) and NSDE (534 images, 134 subjects, and two camera models). The AI algorithm generates the screening report after taking fundus images. The images were labeled as RDR, RMD, GCS, or none of the three by 3 licensed ophthalmologists. The resulting labels were treated as “ground truth” and then were used to compare against the AI screening reports to validate the sensitivity, specificity, and area under the receiver operating characteristic curve (AUC) of the AI algorithm.ResultsOn the PUMCH dataset, regarding the prediction of RDR, the AI algorithm achieved overall results of 0.950 ± 0.058, 0.963 ± 0.024, and 0.954 ± 0.049 on sensitivity, specificity, and AUC, respectively. For RMD, the overall results are 0.919 ± 0.073, 0.929 ± 0.039, and 0.974 ± 0.009. For GCS, the overall results are 0.950 ± 0.059, 0.946 ± 0.016, and 0.976 ± 0.025.ConclusionThe AI algorithm can work robustly with various fundus camera models and achieve high accuracies for detecting RDR, RMD, and GCS.

Artificial intelligence promotes the diagnosis and screening of diabetic retinopathy

“…In China, training and validation data for AI algorithms are vast due to a rather centralized healthcare system and the largest population of DR ( 93 ). Hundreds of new start-up companies working on AI applications to healthcare in China have emerged to improve business, and several DR AI-based screening tools have acquired the certificate of medical device Class III approved by NMPA as pioneers, e.g., Silicon Intelligence, Airdoc, Vistel, and especially Intelligent Healthcare of Baidu that developed the first granted algorithm working robustly with various fundus camera models and achieving high accuracies for detecting multiple ophthalmic diseases ( 94 , 95 ). Undoubtedly, the real-world deployment of these new systems in multiple settings will be full of challenges not only in AI diagnostic technologies but also in the marketing pattern and policy-making.…”

The utilization of artificial intelligence in glaucoma: diagnosis versus screening

“…This involves finding the right balance between sensitivity and specificity, allowing customization based on the specific requirements of screening or diagnosis. The sensitivity-specificity tradeoff refers to the model's ability to correctly identify positives (sensitivity) and negatives (specificity), and tuning helps tailor its performance to the desired emphasis in either screening or diagnostic applications (49). When tuning an AI model for screening, one should prioritize high sensitivity to minimize false negatives, ensuring a better chance of catching potential cases.…”

A 178-clinical-center experiment of integrating AI solutions for lung pathology diagnosis

“…One of the problems of such studies is that algorithms are often tested on the internal datasets sampled from the same source as training images, which makes the results subjected to dataset bias, potential data contamination, and low data representativeness. To mitigate these issues, large-scale multicenter studies have been conducted allowing the researchers to better estimate the prospects of AI for, for example, eye disease diagnosis 11 , 12 . Such studies are, however, expensive and lengthy.…”