Screening Diabetic Retinopathy Using an Automated Retinal Image Analysis System in Independent and Assistive Use Cases in Mexico: Randomized Controlled Trial (Preprint)

Abstract

BACKGROUND The automated screening of patients at risk of developing diabetic retinopathy (DR) represents an opportunity to improve their mid-term outcome, and lower the public expenditure associated with direct and indirect costs of common sight-threatening complications of diabetes. OBJECTIVE The present study, aims to develop and evaluate the performance of an automated deep learning–based system to classify retinal fundus images from international and Mexican patients, as referable and non-referable DR cases. In particular, the performance of the automated retina image analysis (ARIA) system is evaluated under an independent scheme (i.e. only ARIA screening) and two assistive schemes (i.e., hybrid ARIA + ophthalmologist screening), using a web-based platform for remote image analysis to determine and compare the sensibility and specificity of the three schemes. METHODS A randomized controlled experiment was performed where seventeen ophthalmologists were asked to classify a series of retinal fundus images under three different conditions: 1) screening the fundus image by themselves (solo), 2) screening the fundus image after being exposed to the retina image classification of the ARIA system (ARIA answer), and 3) screening the fundus image after being exposed to the classification of the ARIA system, as well as its level of confidence and an attention map highlighting the most important areas of interest in the image according to the ARIA system (ARIA explanation). The ophthalmologists’ classification in each condition and the result from the ARIA system were compared against a gold standard generated by consulting and aggregating the opinion of three retina specialists for each fundus image. RESULTS The ARIA system was able to classify referable vs. non-referable cases with an area under the Receiver Operating Characteristic curve (AUROC) of 98.0% and a sensitivity and specificity of 95.1% and 91.5% respectively, for international patient-cases; and an AUROC, sensitivity, and specificity of 98.3%, 95.2%, and 90.0% respectively for Mexican patient-cases. The results achieved outperformed the average performance of the seventeen ophthalmologists enrolled in the study. Additionally, the achieved results suggest that the ARIA system can be useful as an assistive tool, as significant sensitivity improvements were observed in the experimental condition where ophthalmologists were exposed to the ARIA’s system answer previous to their own classification (93.3%), compared to the sensitivity of the condition where participants assessed the images independently (87.3%). CONCLUSIONS These results demonstrate that both use cases of the ARIA system, independent and assistive, present a substantial opportunity for Latin American countries like Mexico, towards an efficient expansion of monitoring capacity for the early detection of diabetes-related blindness.

Artificial Intelligence and Medicine

“…The DL approach was employed to analyze retinal pictures in order to achieve automatic diabetic retinopathy screening and severity rating (20,21). The AI method's sensitivity and specificity for diagnosing serious lesions were 100% and 88.4%, respectively.…”