Multicenter, Head-to-Head, Real-World Validation Study of Seven Automated Artificial Intelligence Diabetic Retinopathy Screening Systems

Lee, Aaron; Yanagihara, Ryan T.; Lee, Cecilia S.; Blazes, Marian; Jung, Hoon; Chee, Yewlin E.; Gencarella, Michael D.; Gee, Harriet E.; Maa, April Y.; Cockerham, Glenn C.; Lynch, Mary G.; Boyko, Edward J.

doi:10.2337/dc20-1877

Cited by 98 publications

(82 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Therefore, testing on realworld data is required before the clinical use of machine learning systems. 31 To overcome this problem, we adopted the ensemble learning technique and validated the model using unseen datasets, including the prospectively designed dataset and completely independent external dataset. Our proposed method performed well across all datasets, including the Korean and Japanese datasets.…”

Section: Discussionmentioning

confidence: 99%

Development of a Web-Based Ensemble Machine Learning Application to Select the Optimal Size of Posterior Chamber Phakic Intraocular Lens

Kang

Ryu

Lee³

et al. 2021

Trans. Vis. Sci. Tech.

View full text Add to dashboard Cite

Selecting the optimal lens size by predicting the postoperative vault can reduce complications after implantation of an implantable collamer lens with a centralhole (ICL with KS-aquaport). We built a web-based machine learning application that incorporated clinical measurements to predict the postoperative ICL vault and select the optimal ICL size. Methods:We applied the stacking ensemble technique based on eXtreme Gradient Boosting (XGBoost) and a light gradient boosting machine to pre-operative ocular data from two eye centers to predict the postoperative vault. We assigned the Korean patient data to a training (N = 2756 eyes) and internal validation (N = 693 eyes) datasets (prospective validation). Japanese patient data (N = 290 eyes) were used as an independent external dataset from different centers to validate the model. Results:We developed an ensemble model that showed statistically better performance with a lower mean absolute error for ICL vault prediction (106.88 μm and 143.69 μm in the internal and external validation, respectively) than the other machine learning techniques and the classic ICL sizing methods did when applied to both validation datasets. Considering the lens size selection accuracy, our proposed method showed the best performance for both reference datasets (75.9% and 67.4% in the internal and external validation, respectively). Conclusions:Applying the ensemble approach to a large dataset of patients who underwent ICL implantation resulted in a more accurate prediction of vault size and selection of the optimal ICL size.Translational Relevance: We developed a web-based application for ICL sizing to facilitate the use of machine learning calculators for clinicians.

show abstract

Section: Discussionmentioning

confidence: 99%

Development of a Web-Based Ensemble Machine Learning Application to Select the Optimal Size of Posterior Chamber Phakic Intraocular Lens

Kang

Ryu

Lee³

et al. 2021

Trans. Vis. Sci. Tech.

View full text Add to dashboard Cite

show abstract

“…The second prospective, multi-centre comparison study was recently published by Lee and colleagues [ 15 ]. The study compared 7 algorithms from 5 different companies against screening encounters at two different hospitals, for a total of 23,724 patient encounters.…”

Section: Discussionmentioning

confidence: 99%

“…The study compared 7 algorithms from 5 different companies against screening encounters at two different hospitals, for a total of 23,724 patient encounters. The algorithm names, in the aforementioned study’s results, have been anonymised to encourage participation [ 15 ]. Neither IDx-DR nor Retinalyze participated in this comparison.…”

Section: Discussionmentioning

confidence: 99%

“…The performance of different AI algorithms was also compared to a single teleretinal grader that originally graded the given patient, somewhat similarly to the single reader used in this study. The accuracy results varied significantly between the algorithms with only 3 out of 7 achieving comparable sensitivity and one achieving comparable specificity to the original teleretinal graders [ 15 ].…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Analysis and Comparison of Two Artificial Intelligence Diabetic Retinopathy Screening Algorithms in a Pilot Study: IDx-DR and Retinalyze

Grzybowski

Brona²

2021

JCM

View full text Add to dashboard Cite

Background: The prevalence of diabetic retinopathy (DR) is expected to increase. This will put an increasing strain on health care resources. Recently, artificial intelligence-based, autonomous DR screening systems have been developed. A direct comparison between different systems is often difficult and only two such comparisons have been published so far. As different screening solutions are now available commercially, with more in the pipeline, choosing a system is not a simple matter. Based on the images gathered in a local DR screening program we performed a retrospective comparison of IDx-DR and Retinalyze. Methods: We chose a non-representative sample of all referable DR positive screening subjects (n = 60) and a random selection of DR negative patient images (n = 110). Only subjects with four good quality, 45-degree field of view images, a macula-centered and disc-centered image from both eyes were chosen for comparison. The images were captured by a Topcon NW-400 fundus camera, without mydriasis. The images were previously graded by a single ophthalmologist. For the purpose of this comparison, we assumed two screening strategies for Retinalyze—where either one or two out of the four images needed to be marked positive by the system for an overall positive result at the patient level. Results: Percentage agreement with a single reader in DR positive and DR negative cases respectively was: 93.3%, 95.5% for IDx-DR; 89.7% and 71.8% for Retinalyze strategy 1; 74.1% and 93.6% for Retinalyze under strategy 2. Conclusions: Both systems were able to analyse the vast majority of images. Both systems were easy to set up and use. There were several limitations to the current pilot study, concerning sample choice and the reference grading that need to be addressed before attempting a more robust future study.

show abstract

“…In most reported studies, medical AI systems perform excellently in both internal and external validations (Lin et al, 2018;Gurovich et al, 2019;Topol, 2019;Yamashita et al, 2021). However, the performances of AI systems in realworld applications are below expectations with much lower accuracies than the reported results (Lin H. et al, 2019;Baylor et al, 2020;Lee et al, 2021). More valid and exact methods are necessary to verify the effectiveness of the real application of medical AI systems to translate AI into clinical practice more safely (Cabitza et al, 2020;Lin and Yu, 2020).…”

Section: Introductionmentioning

confidence: 95%

Application of Surgical Decision Model for Patients With Childhood Cataract: A Study Based on Real World Data

Chen

Xiang

et al. 2021

Front. Bioeng. Biotechnol.

View full text Add to dashboard Cite

Reliable validated methods are necessary to verify the performance of diagnosis and therapy-assisted models in clinical practice. However, some validated results have research bias and may not reflect the results of real-world application. In addition, the conduct of clinical trials has executive risks for the indeterminate effectiveness of models and it is challenging to finish validated clinical trials of rare diseases. Real world data (RWD) can probably solve this problem. In our study, we collected RWD from 251 patients with a rare disease, childhood cataract (CC) and conducted a retrospective study to validate the CC surgical decision model. The consistency of the real surgical type and recommended surgical type was 94.16%. In the cataract extraction (CE) group, the model recommended the same surgical type for 84.48% of eyes, but the model advised conducting cataract extraction and primary intraocular lens implantation (CE + IOL) surgery in 15.52% of eyes, which was different from the real-world choices. In the CE + IOL group, the model recommended the same surgical type for 100% of eyes. The real-recommended matched rates were 94.22% in the eyes of bilateral patients and 90.38% in the eyes of unilateral patients. Our study is the first to apply RWD to complete a retrospective study evaluating a clinical model, and the results indicate the availability and feasibility of applying RWD in model validation and serve guidance for intelligent model evaluation for rare diseases.

show abstract

Multicenter, Head-to-Head, Real-World Validation Study of Seven Automated Artificial Intelligence Diabetic Retinopathy Screening Systems

Cited by 98 publications

References 32 publications

Development of a Web-Based Ensemble Machine Learning Application to Select the Optimal Size of Posterior Chamber Phakic Intraocular Lens

Development of a Web-Based Ensemble Machine Learning Application to Select the Optimal Size of Posterior Chamber Phakic Intraocular Lens

Analysis and Comparison of Two Artificial Intelligence Diabetic Retinopathy Screening Algorithms in a Pilot Study: IDx-DR and Retinalyze

Application of Surgical Decision Model for Patients With Childhood Cataract: A Study Based on Real World Data

Contact Info

Product

Resources

About