Evaluation of a Deep Learning System For Identifying Glaucomatous Optic Neuropathy Based on Color Fundus Photographs

Abstract

Precis: Pegasus outperformed 5 of the 6 ophthalmologists in terms of diagnostic performance, and there was no statistically significant difference between the deep learning system and the “best case” consensus between the ophthalmologists. The agreement between Pegasus and gold standard was 0.715, whereas the highest ophthalmologist agreement with the gold standard was 0.613. Furthermore, the high sensitivity of Pegasus makes it a valuable tool for screening patients with glaucomatous optic neuropathy. Purpose: The purpose of this study was to evaluate the performance of a deep learning system for the identification of glaucomatous optic neuropathy. Materials and Methods: Six ophthalmologists and the deep learning system, Pegasus, graded 110 color fundus photographs in this retrospective single-center study. Patient images were randomly sampled from the Singapore Malay Eye Study. Ophthalmologists and Pegasus were compared with each other and to the original clinical diagnosis given by the Singapore Malay Eye Study, which was defined as the gold standard. Pegasus’ performance was compared with the “best case” consensus scenario, which was the combination of ophthalmologists whose consensus opinion most closely matched the gold standard. The performance of the ophthalmologists and Pegasus, at the binary classification of nonglaucoma versus glaucoma from fundus photographs, was assessed in terms of sensitivity, specificity and the area under the receiver operating characteristic curve (AUROC), and the intraobserver and interobserver agreements were determined. Results: Pegasus achieved an AUROC of 92.6% compared with ophthalmologist AUROCs that ranged from 69.6% to 84.9% and the “best case” consensus scenario AUROC of 89.1%. Pegasus had a sensitivity of 83.7% and a specificity of 88.2%, whereas the ophthalmologists’ sensitivity ranged from 61.3% to 81.6% and specificity ranged from 80.0% to 94.1%. The agreement between Pegasus and gold standard was 0.715, whereas the highest ophthalmologist agreement with the gold standard was 0.613. Intraobserver agreement ranged from 0.62 to 0.97 for ophthalmologists and was perfect (1.00) for Pegasus. The deep learning system took ∼10% of the time of the ophthalmologists in determining classification. Conclusions: Pegasus outperformed 5 of the 6 ophthalmologists in terms of diagnostic performance, and there was no statistically significant difference between the deep learning system and the “best case” consensus between the ophthalmologists. The high sensitivity of Pegasus makes it a valuable tool for screening patients with glaucomatous optic neuropathy. Future work will extend this study to a larger sample of patients.

Applications of deep learning in detection of glaucoma: A systematic review

“…These results are particularly encouraging as any screening program for glaucoma will need to demonstrate a fairly high specificity to minimize the rate of false positive referrals while ensuring that those with functional vision loss are detected. Thus, these and other studies 25,[27][28][29][30][31][32][33][34][35][36][37][38][39][40][41][42][43][44] have demonstrated that there is a viable potential for medical systems to harness existing teleretinal or other telehealth infrastructure for glaucoma screening if the review of such imaging becomes feasibly automated through DL in the future. Three convolutional layers, one max-pooling layer, and one full connection layer.…”

“…A majority of the studies included in this review utilized CFPs for training and testing the DL algorithm for the diagnosis of glaucoma. 25,[27][28][29][30][31][32][33][34][35][36][37][38][39][40][41][42][43][44] The details of each study with CFPs is summarized in Table 1. Fundus photography has already been successfully incorporated into teleophthalmology programs to detect other eye diseases, such as diabetic retinopathy.…”

“…SDAIRC, AIML, Masker, Mammoth, cvblab, and Winterfell). 25,[29][30][31]33,34,36,38,39,42,44 InceptionNet was used by several studies 27,29,34,37,41 and one entry (cvblab) in the REFUGE challenge. 25 Finally, VGG-Net was used by three studies 29,34,40 and by one of the REF-UGE challenge entries (cvblab).…”

“…Several studies have demonstrated that it is possible to achieve high diagnostic accuracy when an algorithm is tested on images acquired on a different camera than the images used for training. For example, Al-Aswad et al 44 reported an AUC of 0.926, and Liu et al 36 reported an AUC of 0.996 even though the CFPs for testing were not acquired on the same camera as the CFPs used in the training process. If an algorithm is able to show comparable performance on imaging acquired in different cameras, this greatly improves its generalizability and increases the chance that it can be applied in different clinical settings.…”

“…Thus, reporting on the racial/ethnic composition of the dataset used for training and testing a CNN is important to better understanding the generalizability of the findings. However, only a small number of publications 25,29,36,40,44 included racial/ethnic information in their demographic results. For example, Christopher et al 29 reported that 45% of patients in the glaucoma group and 28% of patients in the healthy group were of African compared to European (glaucoma: 50.8%, healthy: 64.2%) or Asian (glaucoma: 5.3%, healthy: 2.1%) descent.…”