Multimodal temporal-clinical note network for mortality prediction

Yang, Haiyang; Kuang, Li; Xia, FengQiang

doi:10.1186/s13326-021-00235-3

Cited by 16 publications

(10 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The two representations were then concatenated to make the predictions. Concurrently to our work, Yang et al (2021) also showed the usefulness of combining time-series data with information from clinical notes. They used an LSTM model for the time-series part of the data and use a convolutional neural network with label-aware attention layer for the clinical notes.…”

Section: Related Worksupporting

confidence: 64%

Predicting in-hospital mortality by combining clinical notes with time-series data

Deznabi¹,

Iyyer²,

Fiterau³

2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

In intensive care units (ICUs), patient health is monitored through (1) continuous vital signals from various medical devices, and (2) clinical notes consisting of opinions and summaries from doctors which are recorded in electronic health records (EHR). It is difficult to jointly model these two sources of information because clinical notes, unlike vital signals, are collected at irregular intervals and their contents are relatively unstructured. In this paper, we present a model that combines both sources of information about ICU patients to make accurate in-hospital mortality predictions. We apply a fine-tuned BERT model to each of the patient's clinical notes. The resulting embeddings are then combined to obtain the overall embedding for the entire text part of the data. This is then combined with the output of an LSTM model that encodes patients' vital signals. Our model improves upon the state of the art for mortality prediction, attaining an AUC score of 0.9, compared to the previous 0.87, setting a new standard for mortality prediction on the MIMIC III benchmark. 1

show abstract

Section: Related Worksupporting

confidence: 64%

Predicting in-hospital mortality by combining clinical notes with time-series data

Deznabi¹,

Iyyer²,

Fiterau³

2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

show abstract

“…Bayesian's methods are typically employed at this level 40 to support a voting process between the set of models into a global decision. Within late fusion there has been headway made to perform multitask deep learning [41][42][43][44][45][46][47] . A schematic for the 3 subtypes of data fusion is presented in Fig.…”

Section: Box 1 Terms and Conceptsmentioning

confidence: 99%

“…was taken by Tang et al who used three-dimensional CNNs and merged the layers in the last layer 113 . EHR and text data were fused together in 11 papers 41,44,80,107,109,116,122,126,134,136,142 . Of these, six 41,44,80,122,134,142 used long term short term (LSTM) networks, CNNs, or knowledge-guided CNNs 160 in their fusion of EHR and clinical notes.…”

Section: Intermediate Fusionmentioning

confidence: 99%

“…EHR and text data were fused together in 11 papers 41,44,80,107,109,116,122,126,134,136,142 . Of these, six 41,44,80,122,134,142 used long term short term (LSTM) networks, CNNs, or knowledge-guided CNNs 160 in their fusion of EHR and clinical notes. Chowdhury et al used graph neural networks and autoencoders to learn meta-embeddings from structured lab test results and clinical notes 107,109 .…”

Section: Intermediate Fusionmentioning

confidence: 99%

See 1 more Smart Citation

Multimodal machine learning in precision health: A scoping review

et al. 2022

View full text Add to dashboard Cite

Machine learning is frequently being leveraged to tackle problems in the health sector including utilization for clinical decision-support. Its use has historically been focused on single modal data. Attempts to improve prediction and mimic the multimodal nature of clinical expert decision-making has been met in the biomedical field of machine learning by fusing disparate data. This review was conducted to summarize the current studies in this field and identify topics ripe for future research. We conducted this review in accordance with the PRISMA extension for Scoping Reviews to characterize multi-modal data fusion in health. Search strings were established and used in databases: PubMed, Google Scholar, and IEEEXplore from 2011 to 2021. A final set of 128 articles were included in the analysis. The most common health areas utilizing multi-modal methods were neurology and oncology. Early fusion was the most common data merging strategy. Notably, there was an improvement in predictive performance when using data fusion. Lacking from the papers were clear clinical deployment strategies, FDA-approval, and analysis of how using multimodal approaches from diverse sub-populations may improve biases and healthcare disparities. These findings provide a summary on multimodal data fusion as applied to health diagnosis/prognosis problems. Few papers compared the outputs of a multimodal approach with a unimodal prediction. However, those that did achieved an average increase of 6.4% in predictive accuracy. Multi-modal machine learning, while more robust in its estimations over unimodal methods, has drawbacks in its scalability and the time-consuming nature of information concatenation.

show abstract

“…Recently, several researchers have modeled both kinds of data for mortality prediction. For example, Yang et al (2021) proposed a multimodal deep neural network that considers both time series data and more clinical records. In addition, chronic and non-chronic patients are classified when dealing with clinical records.…”

Section: Introductionmentioning

confidence: 99%

Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction

Niu

Zhang²,

Peng³

et al. 2023

Front. Mol. Biosci.

View full text Add to dashboard Cite

In intensive care units (ICUs), mortality prediction is performed by combining information from these two sources of ICU patients by monitoring patient health. Respectively, time series data generated from each patient admission to the ICU and clinical records consisting of physician diagnostic summaries. However, existing mortality prediction studies mainly cascade the multimodal features of time series data and clinical records for prediction, ignoring thecross-modal correlation between the underlying features in different modal data. To address theseissues, we propose a multimodal fusion model for mortality prediction that jointly models patients’ time-series data as well as clinical records. We apply a fine-tuned Bert model (Bio-Bert) to the patient’s clinical record to generate a holistic embedding of the text part, which is then combined with the output of an LSTM model encoding the patient’s time-series data to extract valid features. The global contextual information of each modal data is extracted using the improved fusion module to capture the correlation between different modal data. Furthermore, the improved fusion module can be easily added to the fusion features of any unimodal network and utilize existing pre-trained unimodal model weights. We use a real dataset containing 18904 ICU patients to train and evaluate our model, and the research results show that the representations obtained by themodel can achieve better prediction accuracy compared to the baseline.

show abstract

Multimodal temporal-clinical note network for mortality prediction

Cited by 16 publications

References 27 publications

Predicting in-hospital mortality by combining clinical notes with time-series data

Predicting in-hospital mortality by combining clinical notes with time-series data

Multimodal machine learning in precision health: A scoping review

Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction

Contact Info

Product

Resources

About