Evaluating an automated machine learning model that predicts visual acuity outcomes in patients with neovascular age-related macular degeneration

Abstract

Purpose Neovascular age-related macular degeneration (nAMD) is a major global cause of blindness. Whilst anti-vascular endothelial growth factor (anti-VEGF) treatment is effective, response varies considerably between individuals. Thus, patients face substantial uncertainty regarding their future ability to perform daily tasks. In this study, we evaluate the performance of an automated machine learning (AutoML) model which predicts visual acuity (VA) outcomes in patients receiving treatment for nAMD, in comparison to a manually coded model built using the same dataset. Furthermore, we evaluate model performance across ethnic groups and analyse how the models reach their predictions. Methods Binary classification models were trained to predict whether patients’ VA would be ‘Above’ or ‘Below’ a score of 70 one year after initiating treatment, measured using the Early Treatment Diabetic Retinopathy Study (ETDRS) chart. The AutoML model was built using the Google Cloud Platform, whilst the bespoke model was trained using an XGBoost framework. Models were compared and analysed using the What-if Tool (WIT), a novel model-agnostic interpretability tool. Results Our study included 1631 eyes from patients attending Moorfields Eye Hospital. The AutoML model (area under the curve [AUC], 0.849) achieved a highly similar performance to the XGBoost model (AUC, 0.847). Using the WIT, we found that the models over-predicted negative outcomes in Asian patients and performed worse in those with an ethnic category of Other. Baseline VA, age and ethnicity were the most important determinants of model predictions. Partial dependence plot analysis revealed a sigmoidal relationship between baseline VA and the probability of an outcome of ‘Above’. Conclusion We have described and validated an AutoML-WIT pipeline which enables clinicians with minimal coding skills to match the performance of a state-of-the-art algorithm and obtain explainable predictions.

Defining the structure–function relationship of specific lesions in early and advanced age-related macular degeneration

“…With the rapid progress of AI-driven applications, DL models have been developed for the prediction of structure–function correlation in AMD 33 , 34 . Most of these efforts have targeted prediction of VA from OCT images 35 , 36 . Specific to microperimetry, Seebock et al developed a deep learning model (ReSensNet) for the prediction of retinal sensitivities from OCT images with AMD 37 .…”

Clinical performance of automated machine learning: a systematic review

“…Reporting comprehensive metrics is essential, particularly in the context of diagnostic algorithms, as some metrics are a function of prevalence or model threshold (17). Second, explainability analysis is challenging for similar reasons regarding portability, but is possible with emerging technological solutions (22). In addition, some platforms incorporate inbuilt explainability, such as by providing salience maps for DL models.…”

Applications of interpretability in deep learning models for ophthalmology

“…They discovered differing final VA outcome predictions for an 84-year-old British male, and a female of the same age, nationality and initial VA. This indicated the decision boundary for an individual patient level [51].…”