A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy

Beede, Emma; Baylor, Elizabeth; Hersch, Fred; Iurchenko, Anna; Wilcox, Lauren; Ruamviboonsuk, Paisan; Vardoulakis, Laura

doi:10.1145/3313831.3376718

Cited by 347 publications

(290 citation statements)

References 22 publications

Supporting

Mentioning

240

Contrasting

Unclassified

Order By: Relevance

“…Further work is required to understand a holistic interaction and integration of an AI software into clinical workflows within a given health care system. 31…”

Section: Discussionmentioning

confidence: 99%

Evidence Based Prediction and Progression Monitoring on Retinal Images from Three Nations

Turk

Wang

Krause

et al. 2020

Trans. Vis. Sci. Tech.

View full text Add to dashboard Cite

The aim of this work is to demonstrate how a retinal image analysis system, DAPHNE, supports the optimization of diabetic retinopathy (DR) screening programs for grading color fundus photography. Method: Retinal image sets, graded by trained and certified human graders, were acquired from Saudi Arabia, China, and Kenya. Each image was subsequently analyzed by the DAPHNE automated software. The sensitivity, specificity, and positive and negative predictive values for the detection of referable DR or diabetic macular edema were evaluated, taking human grading or clinical assessment outcomes to be the gold standard. The automated software's ability to identify co-pathology and to correctly label DR lesions was also assessed. Results: In all three datasets the agreement between the automated software and human grading was between 0.84 to 0.88. Sensitivity did not vary significantly between populations (94.28%-97.1%) with specificity ranging between 90.33% to 92.12%. There were excellent negative predictive values above 93% in all image sets. The software was able to monitor DR progression between baseline and follow-up images with the changes visualized. No cases of proliferative DR or DME were missed in the referable recommendations. Conclusions: The DAPHNE automated software demonstrated its ability not only to grade images but also to reliably monitor and visualize progression. Therefore it has the potential to assist timely image analysis in patients with diabetes in varied populations and also help to discover subtle signs of sight-threatening disease onset. Translational Relevance: This article takes research on machine vision and evaluates its readiness for clinical use.

show abstract

“…Further work is required to understand a holistic interaction and integration of an AI software into clinical workflows within a given health care system. 31…”

Section: Discussionmentioning

confidence: 99%

Evidence Based Prediction and Progression Monitoring on Retinal Images from Three Nations

Turk

Wang

Krause

et al. 2020

Trans. Vis. Sci. Tech.

View full text Add to dashboard Cite

show abstract

“…Indeed, recent studies provide evidence that tumor immune architecture can greatly dictate clinical efficacy of immune checkpoint inhibitor 56 and poly (ADP-ribose) polymerase (PARP) inhibitor therapies 57 . Lastly, during both model development and evaluation, we sought to emphasize robustness to real-world variability 58 . In particular, we supplemented TCGA WSIs with additional diverse datasets during CNN training, integrated pathologist feedback into model iterations, and evaluated HIF-based model performance on hold-out sets composed exclusively of samples from unseen tissue source sites, improving upon prior approaches to predicting molecular outcomes from TCGA H&E images 22,59 .…”

Section: Discussionmentioning

confidence: 99%

Dense, high-resolution mapping of cells and tissues from pathology images for the interpretable prediction of molecular phenotypes in cancer

Diao

Chui

Wang

et al. 2020

Preprint

View full text Add to dashboard Cite

While computational methods have made substantial progress in improving the accuracy and throughput of pathology workflows for diagnostic, prognostic, and genomic prediction, lack of interpretability remains a significant barrier to clinical integration. In this study, we present a novel approach for predicting clinically-relevant molecular phenotypes from histopathology whole-slide images (WSIs) using human-interpretable image features (HIFs). Our method leverages >1.6 million annotations from board-certified pathologists across >5,700 WSIs to train deep learning models for high-resolution tissue classification and cell detection across entire WSIs in five cancer types. Combining cell- and tissue-type models enables computation of 607 HIFs that comprehensively capture specific and biologically-relevant characteristics of multiple tumors. We demonstrate that these HIFs correlate with well-known markers of the tumor microenvironment (TME) and can predict diverse molecular signatures, including immune checkpoint protein expression and homologous recombination deficiency (HRD). Our HIF-based approach provides a novel, quantitative, and interpretable window into the composition and spatial architecture of the TME.

show abstract

“…These analysis techniques are not new, but have not received sufficient attention in the ML-for-healthcare community 33,45,46 . We believe it is time to adopt methods from health delivery science and health services research to provide honest evaluations of machine learning guided interventions in healthcare 47 and to develop a delivery science for AI interventions in healthcare 48 .…”

Section: Discussionmentioning

confidence: 99%

A framework for making predictive models useful in practice

Jung

Kashyap

Avati

et al. 2020

Preprint

View full text Add to dashboard Cite

Objective To analyze the impact of factors in healthcare delivery on the net benefit of triggering an Advanced Care Planning (ACP) workflow based on predictions of 12-month mortality. Materials and Methods We built a predictive model of 12-month mortality using electronic health record data and evaluated the impact of healthcare delivery factors on the net benefit of triggering an ACP workflow based on the models' predictions. Factors included non-clinical reasons that make ACP inappropriate, limited capacity for ACP, inability to follow up due to patient discharge, and availability of an outpatient workflow to follow up on missed cases. We also quantified the relative benefits of increasing capacity for inpatient ACP versus outpatient ACP. Results Work capacity constraints and discharge timing can significantly reduce the net benefit of triggering the ACP workflow based on a model's predictions. However, the reduction can be mitigated by creating an outpatient ACP workflow. Given limited resources to either add capacity for inpatient ACP versus developing outpatient ACP capability, the latter is likely to provide more benefit to patient care. Discussion The benefit of using a predictive model for identifying patients for interventions is highly dependent on the capacity to execute the workflow triggered by the model. We provide a framework for quantifying the impact of healthcare delivery factors and work capacity constraints on achieved benefit. Conclusion An analysis of the sensitivity of the net benefit realized by a predictive model triggered clinical workflow to various healthcare delivery factors is necessary for making predictive models useful in practice.

show abstract

A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy

Cited by 347 publications

References 22 publications

Evidence Based Prediction and Progression Monitoring on Retinal Images from Three Nations

Evidence Based Prediction and Progression Monitoring on Retinal Images from Three Nations

Dense, high-resolution mapping of cells and tissues from pathology images for the interpretable prediction of molecular phenotypes in cancer

A framework for making predictive models useful in practice

Contact Info

Product

Resources

About