Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs

Abstract

IMPORTANCEDeep learning is a family of computational methods that allow an algorithm to program itself by learning from a large set of examples that demonstrate the desired behavior, removing the need to specify rules explicitly. Application of these methods to medical imaging requires further assessment and validation.OBJECTIVE To apply deep learning to create an algorithm for automated detection of diabetic retinopathy and diabetic macular edema in retinal fundus photographs. DESIGN AND SETTINGA specific type of neural network optimized for image classification called a deep convolutional neural network was trained using a retrospective development data set of 128 175 retinal images, which were graded 3 to 7 times for diabetic retinopathy, diabetic macular edema, and image gradability by a panel of 54 US licensed ophthalmologists and ophthalmology senior residents between May and December 2015. The resultant algorithm was validated in January and February 2016 using 2 separate data sets, both graded by at least 7 US board-certified ophthalmologists with high intragrader consistency.EXPOSURE Deep learning-trained algorithm. MAIN OUTCOMES AND MEASURESThe sensitivity and specificity of the algorithm for detecting referable diabetic retinopathy (RDR), defined as moderate and worse diabetic retinopathy, referable diabetic macular edema, or both, were generated based on the reference standard of the majority decision of the ophthalmologist panel. The algorithm was evaluated at 2 operating points selected from the development set, one selected for high specificity and another for high sensitivity. RESULTSThe EyePACS-1 data set consisted of 9963 images from 4997 patients (mean age, 54.4 years; 62.2% women; prevalence of RDR, 683/8878 fully gradable images [7.8%]); the Messidor-2 data set had 1748 images from 874 patients (mean age, 57.6 years; 42.6% women; prevalence of RDR, 254/1745 fully gradable images [14.6%]). For detecting RDR, the algorithm had an area under the receiver operating curve of 0.991 (95% CI, 0.988-0.993) for EyePACS-1 and 0.990 (95% CI, 0.986-0.995) for Messidor-2. Using the first operating cut point with high specificity, for EyePACS-1, the sensitivity was 90.3% (95% CI, 87.5%-92.7%) and the specificity was 98.1% (95% CI, 97.8%-98.5%). For Messidor-2, the sensitivity was 87.0% (95% CI, 81.1%-91.0%) and the specificity was 98.5% (95% CI, 97.7%-99.1%). Using a second operating point with high sensitivity in the development set, for EyePACS-1 the sensitivity was 97.5% and specificity was 93.4% and for Messidor-2 the sensitivity was 96.1% and specificity was 93.9%. CONCLUSIONS AND RELEVANCEIn this evaluation of retinal fundus photographs from adults with diabetes, an algorithm based on deep machine learning had high sensitivity and specificity for detecting referable diabetic retinopathy. Further research is necessary to determine the feasibility of applying this algorithm in the clinical setting and to determine whether use of the algorithm could lead to improved care and outcomes compared with curren...

Status and Direction of Healthcare Data in Korea for Artificial Intelligence

“…Currently, deep learning techniques are intensively applied to the medical image data in the PACS [3,4], because deep learning methods have demonstrated their capabilities for image analysis in other areas. In addition, there are standards for image data, e.g., DICOM or JPEG.…”

“…Recently, artificial intelligence (AI) has been highlighted in various areas including healthcare [1][2][3][4]. AI can be categorized into symbolic AI such as expert systems and machine learning (ML), which includes deep learning.…”

“…We need well-curated and labeled data. For example, 54 US licensed ophthalmologists and ophthalmology senior residents have reviewed 128,175 retinal images to build a well-curated dataset [3]. Current digitalized medical records require more in-depth curation to be used for research.…”

Deep Learning for Medical Image Analysis: Applications to Computed Tomography and Magnetic Resonance Imaging

“…Particularly, after a team using a CNN based model won the ImageNet competition in 2012 by a significant performance gap and machines exceeded humans at an indirect comparison of image recognition task in 2015, the possibility of clinical implementation of deep learning became a major issue [4]. Similarly, in medical imaging, after a CNN based model won the mitotic cell detection task in breast biopsies at the 2012 ICPR (International conference on pattern recognition), recent studies on diabetic retinopathy detection and skin cancer classification demonstrated that deep learning models trained with massive medical images can even surpass the performance of human specialist in diagnostic image analysis [5,6,7].…”

Auspicious machine learning

“…Naturally, neural networks are hungry for training datathe bigger the dataset, the better they will perform when presented with new data. For example, about 130,000 clinical images of skin lesions 1 and of retinal images 2 were recently used to train convolution networks to classify skin cancers 1 and detect diabetic retinopathy 2 , with accuracy comparable to expert dermatologists and ophthalmologists. Yet, can convolutional networks help with rare diseases, for which data are scarce?…”