A Multi-Headed Transformer Approach for Predicting the Patient’s Clinical Time-Series Variables From Charted Vital Signs

Abstract

Deep learning has progressively been the spotlight of innovations that aim to leverage the clinical time-series data that are longitudinally recorded in the Electronic Health Records(EHR) to forecast the patient's survival and vital signs deterioration. However, their recording velocity, as well as their noisiness, hinder the proper adoption of the recently proposed benchmarks. The Recurrent Neural Networks (RNN) especially the Long-short Term Memory (LSTMs) have achieved better results in recent studies but they are hard to train and interpret and fail to properly capture the long-term dependencies. Moreover, the RNNs suffer greatly with clinical time series due to their sequential processing which cripples the prospect of parallel processing. Recently the Transformer approach was proposed for Natural Language Processing (NLP) tasks and achieved state-of-the-art results. Hence to tackle the drawbacks that are suffered by the RNNs we propose a clinical time series Multi-head Transformer (MHT), which is a transformerbased model that forecasts the patient's future time series variables using the vitals signs. To prove the generalization of the model we use the same model for other critical tasks that describe the Intensive Care Unit (ICU) patient's progression and the associated risks like the remaining Length Of Stay(LoS), the Inhospital Mortality as well as the 24 hours mortality. Our model achieves an Area Under The Curve-Receiver Operating Characteristics( AUC-ROC) of 0.98 and an Area Under the Curve, Precision-Recall (AUC-PR) of 0.424 for vital time series prediction, and an AUC-ROC of 0.875 in the mortality prediction. The model performs well for the frequently recorded variables like the Heart Rate (HR) and performs barely like the LSTM counterparts for the intermittently captured records such as the White Blood Count (WBC).

Generative AI models in time varying biomedical data: a systematic review (Preprint)

“…Extending from a single prediction point, this work uses LSTM to model irregularly timed time series data and predict multistep future trajectories on diabetes and mental health cohorts [74]. In recent years, researchers have also turned to transformer models for prediction tasks [75][76][77] as they capture more long-range dependencies and provide more interpretability of model weights than RNNs. In this work, selfattention is used to measure each feature's relevance to each other and also each time point's relevance within each feature to predict delirium in critical care [78], and another paper uses similar mechanisms to predict cardiac patient mortality risk [79].…”

Design of an Iterative Method for Enhanced Multimodal Time Series Analysis Using Graph Attention Networks, Variational Graph Autoencoders, and Transfer Learning

“…These adaptations, as explored by and Na et al (2023), underscore the ongoing efforts to refine ESNs for better performance. The incorporation of attention mechanisms and transformers, as employed in the multi-headed transformer approach by Harerimana et al (2022), represents another pivotal trend. These methods, rooted in natural language processing, have shown promising results in capturing long-term dependencies and enhancing the interpretability of predictions, marking a significant leap forward in the analysis of clinical and multivariate time series.…”