How to Leverage Multimodal EHR Data for Better Medical Predictions?

Abstract

Healthcare is becoming a more and more important research topic recently. With the growing data in the healthcare domain, it offers a great opportunity for deep learning to improve the quality of medical service. However, the complexity of electronic health records (EHR) data is a challenge for the application of deep learning. Specifically, the data produced in the hospital admissions are monitored by the EHR system, which includes structured data like daily body temperature, and unstructured data like free text and laboratory measurements. Although there are some preprocessing frameworks proposed for specific EHR data, the clinical notes that contain significant clinical value are beyond the realm of their consideration. Besides, whether these different data from various views are all beneficial to the medical tasks and how to best utilize these data remain unclear. Therefore, in this paper, we first extract the accompanying clinical notes from EHR and propose a method to integrate these data, we also comprehensively study the different models and the data leverage methods for better medical task prediction. The results on two medical prediction tasks show that our fused model with different data outperforms the state-of-the-art method that without clinical notes, which illustrates the importance of our fusion method and the value of clinical note features. Our code is available at https: //github.com/emnlp-mimic/mimic.

Research on Multimodal Fusion of Temporal Electronic Medical Records

“…This flexibility enables the model to better adapt to the characteristics of different tasks or different samples, thereby improving the generalization performance of the model. Yang et al took the lead in introducing the multimodal fusion MAG method [27] into the medical field and proved its effectiveness on multimodal EMR data. Compared with other fusion methods, fusion using MAG can adapt to different task requirements and data characteristics by adjusting the selection of main modalities, and the gating mechanism contained in it can effectively remove redundancy in the data.…”

“…Time series data directly capture the dynamic health trends and treatment outcomes of patients [15,48]. Scholars have proposed that incorporating time series data into prediction models can enhance their performance [27,41,48]. Currently, RNN and their variants are commonly employed in handling medical time series data, as they are well-suited for sequential and time-related tasks [49,50].…”

“…This model employs four encoders to handle four types of patient EMR data, specifically: temporal tabular information, temporal notes, static tabular information, and static notes. The multimodal adaptation gate (MAG) [27] is then employed to effectuate multimodal fusion, with the addition of an attention-backtracking module to enhance the model's capacity to address long-term dependencies. An overview of the multimodal fusion model we propose, and its application, are depicted in Figure 1.…”

Deep Learning for Medical Prediction in Electronic Health Records

“…The complex irregular temporal property and multimodal structure make modeling EHRs for medical predictions challenging. Prior works (Deznabi, Iyyer, and Fiterau 2021;Yang and Wu 2021) ignore the irregularity in each modality in multimodal fusion. To full integrate irregularity into multimodal representation learning, we propose to separately tackle irregularity in each single modality and fuse their representations across temporal steps to improve medical predictions.…”

Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI)

“…Also, most clinical machine learning systems focus on one clinical prediction task at a time (D 'Costa et al, 2020;. However, in real-world systems more than one such task are often performed simultaneously and are interrelated (Yang and Wu, 2021). There is a need to investigate task-specific unified representations of multimodal clinical data in both single-task and multi-task settings to improve decisions in the clinical workflow by demonstrating an increase in predictive power, robustness, and confidence over any single mode of data (Tiulpin et al, 2019).…”