LSAN: Modeling Long-term Dependencies and Short-term Correlations with Hierarchical Attention for Risk Prediction

Abstract

Risk prediction using electronic health records (EHR) is a challenging data mining task due to the two-level hierarchical structure of EHR data. EHR data consist of a set of time-ordered visits, and within each visit, there is a set of unordered diagnosis codes. Existing approaches focus on modeling temporal visits with deep neural network (DNN) techniques. However, they ignore the importance of modeling diagnosis codes within visits, and a lot of task-unrelated information within visits usually leads to unsatisfactory performance of existing approaches. To minimize the effect caused by noise information of EHR data, in this paper, we propose a novel DNN for risk prediction termed as LSAN, which consists of a Hierarchical Attention Module (HAM) and a Temporal Aggregation Module (TAM). Particularly, LSAN applies HAM to model the hierarchical structure of EHR data. Using the attention mechanism in the hierarchy of diagnosis code, HAM is able to retain diagnosis details and assign flexible attention weights to different diagnosis codes by their relevance to corresponding diseases. Moreover, the attention mechanism in the hierarchy of visit learns a comprehensive feature throughout the visit history by paying greater attention to visits with higher relevance. Based on the foundation laying by HAM, TAM uses a two-pathway structure to learn a robust temporal aggregation mechanism among all visits for LSAN. It extracts long-term dependencies by a Transformer encoder and short-term correlations by a parallel convolutional layer among different visits. With the construction of HAM and TAM, LSAN achieves the stateof-the-art performance on three real-world datasets with larger AUCs, recalls and F1 scores. Furthermore, the model analysis results demonstrate the effectiveness of the network construction with good interpretability and robustness of decision making by LSAN 1 .

MeSIN: Multilevel Selective and Interactive Network for Medication Recommendation

“…For instance, GRAM [7], KAME [25], and G-BERT [30] leveraged the attention mechanism to integrate domain knowledge into disease or medication code representations for better performance. Retain [8], Dipole [24], Timeline [3] and LSAN [35] all introduced attention mechanism to model the disease progression by considering the dependencies among visits and provide some interpretable insights. In addition, GCT [10] were equipped with advanced attention networks, i.e., Transformer [33], to build the correlations between medical codes from every visits based on the automatically learned attention weights.…”

“…1, EHR systems in hospitals accumulate complex temporal and heterogeneous sequences. Existing studies in health informatics have widely utilize the temporal sequential records from EHRs to solve healthcare problems such as predicting disease progression [9], [29], [40], [35], medications recommendation [38], [31], [18] and clinical trial recruitment [5], [37]. However, most of the studies such as T-LSTM [4], MNN [29] and LSAN [35] mainly focused on modelling the temporal dependencies of multiple visits of homogeneous sequence such as the history diseases sequence.…”

“…Existing studies in health informatics have widely utilize the temporal sequential records from EHRs to solve healthcare problems such as predicting disease progression [9], [29], [40], [35], medications recommendation [38], [31], [18] and clinical trial recruitment [5], [37]. However, most of the studies such as T-LSTM [4], MNN [29] and LSAN [35] mainly focused on modelling the temporal dependencies of multiple visits of homogeneous sequence such as the history diseases sequence. While the medication recommendation task involves multiple temporal and heterogeneous sequences, not only the temporal intradependencies but also the inter-correlations between the sequences should be considered when modelling the sequence learning process.…”

“…Third, unlike the above discussed structure-related limitations, how to recognize and filter out noisy information existing in EHR data at each timestamp is another important challenge that inhibits the recommendation performance. However, few deep learning studies in health informatics focus on infusing the feature selections into the learning process except LSAN [35] which considers assigning flexible attention weights to different diagnosis codes via their relevance to corresponding diseases for reducing the effect of irrelevant diagnosis codes in EHR data. However, the doctors practically pay more attention on the critical few factors and neglect the irrelevant medical indicators or historical medical codes.…”

Cardiac Complication Risk Profiling for Cancer Survivors via Multi-View Multi-Task Learning

“…• LSAN[27]. An attention-based model that uses Transformer to capture global information and CNN to capture local information.• MTL Models: We develop the MTL version for each of the aforementioned neural network-based models by employing task-specific attention and decoder over the output generated by these models.…”