Deep Learning Interpretation of Echocardiograms

Abstract

Echocardiography uses ultrasound technology to capture high temporal and spatial resolution images of the heart and surrounding structures and is the most common imaging modality in cardiovascular medicine. Using convolutional neural networks on a large new dataset, we show that deep learning applied to echocardiography can identify local cardiac structures, estimate cardiac function, and predict systemic phenotypes that modify cardiovascular risk but not readily identifiable to human interpretation. Our deep learning model, EchoNet, accurately identified the presence of pacemaker leads (AUC = 0.89), enlarged left atrium (AUC = 0.85), normal left ventricular wall thickness (AUC = 0.75), left ventricular end systolic and diastolic volumes(R 2 = 0.73 and R 2 = 0.68), and ejection fraction (R 2 = 0.48) as well as predicted systemic phenotypes of age (R 2 = 0.46), sex (AUC = 0.88), weight (R 2 = 0.56), and height (R 2 = 0.33). Interpretation analysis validates that EchoNet shows appropriate attention to key cardiac structures when performing human-explainable tasks and highlight hypothesis-generating regions of interest when predicting systemic phenotypes difficult for human interpretation. Machine learning on echocardiography images can streamline repetitive tasks in the clinical workflow, standardize interpretation in areas with insufficient qualified cardiologists, and more consistently produce echocardiographic measurements.Related works Current literature have already shown that it is possible to identify standard echocardiogram views from unlabeled datasets. 5,6,24 Previous works have used convolutional neural networks (CNNs) trained on images and videos from echocardiography to perform segmentation to identify cardiac structures and derive cardiac function. In this study, we extend previous analyses to show that EchoNet, our deep learning model using echocardiography images, local cardiac structures and anatomy, estimate volumetric measurements and metrics of cardiac function, and predict systemic human phenotypes that modify cardiovascular risk. Additionally, we show the first application of interpretation frameworks to understand deep learning models from echocardiogram images. Human identifiable features, such as the presence of pacemaker and defibrillator leads, left ventricular hypertrophy, and abnormal left atrial chamber size identified by our convolutional neural network were validated using interpretation frameworks to highlight the most relevant regions of interest. To the best of our knowledge, we develop the first deep learning model that can directly predict age, sex, weight and height from echocardiogram images and use interpretation methods to understand how the model predicts these systemic phenotypes difficult for human interpreters. 2/14Lessons from model training and experiments EchoNet performance greatly improved with efforts to augment data size, homogenize input data, and with optimize model training with hyperparameter search. Our experience shows that increasing number of unique patien...

Impact of incorporating echocardiographic screening into a clinical prediction model to optimise utilisation of echocardiography in primary care

“…Additionally, medical regulations need to incorporate task shifting of echo acquisition—a debatable topic in Brazil. Also in this context, embedded apps for optimal probe positioning 26 and to flag abnormalities 27 must be warranted for future handheld devices, to minimise practical limitations. Finally, this novel approach for rationalisation of heath resources for cardiology tests deserves further exploration, with validation in different settings and cost‐effectiveness assessment.…”

Introduction to artificial intelligence in ultrasound imaging in obstetrics and gynecology

“…and diagnosis ('what is wrong with the imaged object?'). Active academic research and emerging examples of AI-assisted applications for ultrasound include plane-finding (navigation) and automated quantification for analysis of the breast, prostate, liver and heart [40][41][42] . In obstetric and gynecological ultrasound, promising workload-changing advancements include automatic detection of standard planes and quality assurance in fetal ultrasound [43][44][45] , detection of endometrial thickness in gynecology 46 and automatic classification of ovarian cysts (Table 1).…”

Diagnosis and Risk Prediction of Dilated Cardiomyopathy in the Era of Big Data and Genomics

“…Phenotype-classifying algorithms may be able to cope with conflicting or missing data by combining multiple data sources and integrating information on treatment and comorbidities to infer diagnoses, as shown by a case study in atrial fibrillation [ 89 ]. Furthermore, clinically repetitive or administrative tasks can be automated [ 90 ]—for instance, EchoNet is a deep learning network that can accurately extract LV volume and function, and other algorithms have been deployed for automatic CMR multi-structure segmentation [ 91 , 92 ]. Registration of diagnoses can also be automated, for example by interpreting clinical discharge letters and extracting diagnoses using deep learning [ 93 , 94 ].…”