A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis

Abstract

Background Deep learning offers considerable promise for medical diagnostics. We aimed to evaluate the diagnostic accuracy of deep learning algorithms versus health-care professionals in classifying diseases using medical imaging.Methods In this systematic review and meta-analysis, we searched Ovid-MEDLINE, Embase, Science Citation Index, and Conference Proceedings Citation Index for studies published from Jan 1, 2012, to June 6, 2019. Studies comparing the diagnostic performance of deep learning models and health-care professionals based on medical imaging, for any disease, were included. We excluded studies that used medical waveform data graphics material or investigated the accuracy of image segmentation rather than disease classification. We extracted binary diagnostic accuracy data and constructed contingency tables to derive the outcomes of interest: sensitivity and specificity. Studies undertaking an out-of-sample external validation were included in a meta-analysis, using a unified hierarchical model. This study is registered with PROSPERO, CRD42018091176.Findings Our search identified 31 587 studies, of which 82 (describing 147 patient cohorts) were included. 69 studies provided enough data to construct contingency tables, enabling calculation of test accuracy, with sensitivity ranging from 9•7% to 100•0% (mean 79•1%, SD 0•2) and specificity ranging from 38•9% to 100•0% (mean 88•3%, SD 0•1). An out-of-sample external validation was done in 25 studies, of which 14 made the comparison between deep learning models and health-care professionals in the same sample. Comparison of the performance between health-care professionals in these 14 studies, when restricting the analysis to the contingency table for each study reporting the highest accuracy, found a pooled sensitivity of 87•0% (95% CI 83•0-90•2) for deep learning models and 86•4% (79•9-91•0) for health-care professionals, and a pooled specificity of 92•5% (95% CI 85•1-96•4) for deep learning models and 90•5% (80•6-95•7) for health-care professionals.Interpretation Our review found the diagnostic performance of deep learning models to be equivalent to that of health-care professionals. However, a major finding of the review is that few studies presented externally validated results or compared the performance of deep learning models and health-care professionals using the same sample. Additionally, poor reporting is prevalent in deep learning studies, which limits reliable interpretation of the reported diagnostic accuracy. New reporting standards that address specific challenges of deep learning could improve future studies, enabling greater confidence in the results of future evaluations of this promising technology.

Insights into Systemic Disease through Retinal Imaging-Based Oculomics

“…In a recent systematic review evaluating 82 studies comparing the performance of deep learning models with health care professionals in disease classification from medical imaging, 25 studies leveraged data from open-access repositories. 57 In the field of oculomics, retrospective real-world data can be a more challenging option as the desired labels (such as myocardial infarction within five years) will not align with the original purpose for capture, typically being eye disease. Moreover, ophthalmic care is often provided in standalone ophthalmic settings.…”

Surgery in the 2020s: Implications of advancing technology for patients and the workforce

“…Similarly, the algorithms can exclude 93% of patients who don't have disease, compared to 91% among healthcare professionals. 9 The implications for hospitals are clear. Where a patient might previously have had to visit a specialist for a diagnosis, their tests will instead be able to be analysed locally and a diagnosis delivered instantly.…”

Metalearning as One of the Task of the Machine Learning Problems

“…In [12], a systematic review and meta-analysis were carried out to diagnose any symptom of the disease using medical imaging and histopathology materials, and the accuracy of diagnosing Machine learning algorithms is used for visual recognition. A model of deep learning was created, created thanks to advances in the architecture of parallel computing, which made an important breakthrough in the competition of large-scale visual recognition.…”