A foundational vision transformer improves diagnostic performance for electrocardiograms

Abstract

The electrocardiogram (ECG) is a ubiquitous diagnostic modality. Convolutional neural networks (CNNs) applied towards ECG analysis require large sample sizes, and transfer learning approaches for biomedical problems may result in suboptimal performance when pre-training is done on natural images. We leveraged masked image modeling to create a vision-based transformer model, HeartBEiT, for electrocardiogram waveform analysis. We pre-trained this model on 8.5 million ECGs and then compared performance vs. standard CNN architectures for diagnosis of hypertrophic cardiomyopathy, low left ventricular ejection fraction and ST elevation myocardial infarction using differing training sample sizes and independent validation datasets. We find that HeartBEiT has significantly higher performance at lower sample sizes compared to other models. We also find that HeartBEiT improves explainability of diagnosis by highlighting biologically relevant regions of the EKG vs. standard CNNs. Domain specific pre-trained transformer models may exceed the classification performance of models trained on natural images especially in very low data regimes. The combination of the architecture and such pre-training allows for more accurate, granular explainability of model predictions.

Quantitative Prediction of Right Ventricular Size and Function From the ECG

“…Our group has demonstrated that CNN architectures continue to increase performance with increasing amounts of ECG data even above 500 000 training examples. 18 Given these prior works, we did not seek to demonstrate the additive value of UKBB data, which would require multiple model retrainings at different training sample sizes. This is an interesting direction of research in the data science realm and would be useful to establish minimum sample size estimates for neural network training in the ECG analysis.…”

“…Neural network architectures are uniquely able to extract information‐rich features from complex waveform data and form the basis of several published DL‐ECG models. 14 , 15 , 16 , 17 , 18 We trained the network using the Adam optimizer with cross‐entropy as the loss function for classification and mean absolute error for regression tasks, respectively. To minimize overfitting, we monitored train‐test loss over training epochs against an internal validation set of 5% of ECGs within the training cohort.…”

From Theoretical Models to Practical Deployment: A Perspective and Case Study of Opportunities and Challenges in AI-driven Cardiac Auscultation Research for Low-Income Settings

“…Another avenue worth exploring is the replacement of the Fourier method in spectrogram creation with a signature-based approach [90]. The application of self-supervised learning to incorporate more domain-specific data shows promise, as evidenced by a recent paper that introduced HeartBEiT, a vision-based transformer model for ECG analysis [91]. HeartBEiT demonstrated significantly superior performance at lower sample sizes compared to standard CNNs.…”

Comparative analysis of Attention-based CNN andViT for processing ECG image data in joint fusion

“…Finally, the model leverages both spatial understanding of features from image data and features from clinical data using a fusion layer for a more holistic feature representation. Recently, the Vision Transformer (ViT) model [12] was developed for processing ECG for diagnosis of hypertrophic cardiomyopathy, low left ventricular ejection fraction, and ST elevation myocardial infarction and achieved state-of-the-art performance [12]. For comparative benchmarking on the PCI dataset, we evaluated the predictive performance of the CNN model with block attention which preserves local and semi-global dependencies, against a Vision Transformer (ViT) model which preserves long-range dependency, presenting an extensive analysis of their respective results in the context of post-PCI prognosis.…”