Protocol for a systematic review and meta-analysis of the diagnostic accuracy of artificial intelligence for grading of ophthalmology imaging modalities

Abstract

Background With the rise of artificial intelligence (AI) in ophthalmology, the need to define its diagnostic accuracy is increasingly important. The review aims to elucidate the diagnostic accuracy of AI algorithms in screening for all ophthalmic conditions in patient care settings that involve digital imaging modalities, using the reference standard of human graders. Methods This is a systematic review and meta-analysis. A literature search will be conducted on Ovid MEDLINE, Ovid EMBASE, and Wiley Cochrane CENTRAL from January 1, 2000, to December 20, 2021. Studies will be selected via screening the titles and abstracts, followed by full-text screening. Articles that compare the results of AI-graded ophthalmic images with results from human graders as a reference standard will be included; articles that do not will be excluded. The systematic review software DistillerSR will be used to automate part of the screening process as an adjunct to human reviewers. After the full-text screening, data will be extracted from each study via the categories of study characteristics, patient information, AI methods, intervention, and outcomes. Risk of bias will be scored using Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) by two trained independent reviewers. Disagreements at any step will be addressed by a third adjudicator. The study results will include summary receiver operating characteristic (sROC) curve plots as well as pooled sensitivity and specificity of artificial intelligence for detection of any ophthalmic conditions based on imaging modalities compared to the reference standard. Statistics will be calculated in the R statistical software. Discussion This study will provide novel insights into the diagnostic accuracy of AI in new domains of ophthalmology that have not been previously studied. The protocol also outlines the use of an AI-based software to assist in article screening, which may serve as a reference for improving the efficiency and accuracy of future large systematic reviews. Trial registration PROSPERO, CRD42021274441

Taxonomy of Quality Assessment for Intelligent Software Systems: A Systematic Literature Review

“…QA1 QA2 QA3 QA4 Total P1 Abdellatif et al [16] 1 0 0 1 2 P2 Khan et al [17] 1 1 1 0.5 3.5 P3 Shaikh et al [18] 0 1.5 1 0 2.5 P4 Zarembo et al [19] 0.5 0.5 0.5 0.5 2 P5 Cao et al [20] 1 0.5 0 0 1.5 P6 Yang et al [21] 1 0.5 1 0.5 3 P7 Cote et al [22] 1 1 1 1 4 P8 Xu et al [23] 0.5 1 1 0.5 3 P9 Dey et al [24] 1 1 0.5 0 2.5 P10 Zanca et al [25] 1 0 1 1 3 P11 Al-Dasuqi et al [26] 0.5 1 0.5 0.5 2.5 P12 O'Hare et al [27] 1 1 1 0.5 3.5 P13 Druffel et al [28] 0 0 1 1 2 P14 Perkusich et al [29] 0.5 1 1 1 3.5 P15 Vinsard et al [30] 0.5 0 0.5 1 2 P16 Beegle et al [31] 1 0 1 0.5 2.5 P17 Salahirad et al [32] 1 1 0.5 0.5 3 P18 Dlamini et al [33] 0.5 0.5 0.5 0.5 2 P19 Cross et al [34] 0.5 1 1 1 3.5 P20 Malamateniou et al [35] 0 0 1 1 2 P21 Lenarduzzi et al [7] 1 0.5 0.5 0.5 2.5 P22 Sikorska et al [36] 0 1 0 1 2 P23 Field et al [37] 1 1 0 1 3 P24 Gao et al [38] 0.5 0 0 0 0.5 P25 Khanagar et al [39] 1 0 0 1 2 P26 Shafiq et al [40] 1 0.5 1 1 3.5 P27 Waade et al [41] 1 1 1 0.5 3.5 P28 Felderer et al [42] 1 0 0 0 1 P29 Ji et al [43] 0 0 1 1 2 P30 Harman et al [44] 1 0.5 1 1 3.5 P31 Shehab et al [45] 0 0.5 0.5 0.5 1.5 P32 Rana et al [46] 1 1 1 0 3 P33 Huang et al [47] 1 0.5 0.5 0.5 2.5 P34 Shakeel et al [48] 1 0.5 1 1 3.5 P35 Zhou et al [49] 0.5 0.5 0 0 1 P36 Sayago-Heredia et al…”

Untitled

“…Therefore, to evaluate an AI application, it is essential to use evidence-based medicine principles -a standard that is only sometimes met [26] . Ethical Aspects, Translation, and Conclusion Given the rise of AI in medicine and ophthalmology, defining its accuracy and reliability will guide future research in this area and enhance its real-life adaption [27] . The potential challenges with DL application in ophthalmology include clinical and technical challenges, data quality, explain ability of the algorithm results, medicolegal issues, and physician and patient acceptance of the AI "black box" algorithms.…”

Artificial intelligence in ophthalmology

“…Therefore, to evaluate an AI application, it is essential to use evidence-based medicine principles -a standard that is only sometimes met [26] . Ethical Aspects, Translation, and Conclusion Given the rise of AI in medicine and ophthalmology, defining its accuracy and reliability will guide future research in this area and enhance its real-life adaption [27] . The potential challenges with DL application in ophthalmology include clinical and technical challenges, data quality, explain ability of the algorithm results, medicolegal issues, and physician and patient acceptance of the AI "black box" algorithms.…”