ATTAIN: Attention-based Time-Aware LSTM Networks for Disease Progression Modeling

Abstract

Modeling patient disease progression using Electronic Health Records (EHRs) is critical to assist clinical decision making. Long-Short Term Memory (LSTM) is an effective model to handle sequential data, such as EHRs, but it encounters two major limitations when applied to EHRs: it is unable to interpret the prediction results and it ignores the irregular time intervals between consecutive events. To tackle these limitations, we propose an attention-based time-aware LSTM Networks (ATTAIN), to improve the interpretability of LSTM and to identify the critical previous events for current diagnosis by modeling the inherent time irregularity. We validate ATTAIN on modeling the progression of an extremely challenging disease, septic shock, by using real-world EHRs. Our results demonstrate that the proposed framework outperforms the state-of-the-art models such as RETAIN and T-LSTM. Also, the generated interpretative time-aware attention weights shed some lights on the progression behaviors of septic shock.

Asthma Exacerbation Prediction and Risk Factor Analysis Based on a Time-Sensitive, Attentive Neural Network: Retrospective Cohort Study

“…We built 2 cohorts as simulations for both the real-world application scenario ( early prediction ) and the model evaluation (called next-visit prediction in many previous studies [ 27 , 32 ]). In early prediction, we could not foresee when the exacerbation would happen but could only evaluate the future risk at each visit.…”

“…One of the most popular architectures is recurrent neural networks (RNNs), which make predictions according to the sequence of historical events. Dozens of successes have been achieved in applying deep learning to disease predictions [ 24 ], mostly using variants of RNNs with distinct network components, for example, by adding an attention mechanism to evaluate the weights of each variable [ 25 - 29 ] or by using special configurations to tackle the problem of time decays [ 23 , 25 , 27 , 30 - 32 ]. Typical prediction tasks include the prediction of diabetes mellitus [ 23 ], Parkinson disease [ 29 , 33 ], chronic heart failure [ 26 ], sepsis [ 34 ], mortality, and readmission [ 25 ].…”

“…There are major differences between TSANN and recent state-of-the-art deep learning–based clinical predictive models such as time-aware LSTM (TLSTM) [ 23 ], Reverse Time Attention model (RETAIN) [ 27 ], and Attention-based Time-aware Disease Progression (ATTAIN) [ 32 ]. First, the model structures are different.…”

Asthma Exacerbation Prediction and Risk Factor Analysis Based on a Time-Sensitive, Attentive Neural Network: Retrospective Cohort Study (Preprint)

“…It also uses a similar time decay function as RETAIN. We implemented it ourselves using TensorFlow ATTAIN g [32] The proposed TSANN model but with the second attention layer removed. Prediction is based on the final state of LSTM TSANN h -I Apply the time-encoding method from Song et al [39] on TSANN-I.…”

“…One of the most popular architectures is recurrent neural networks (RNNs), which make predictions according to the sequence of historical events. Dozens of successes have been achieved in applying deep learning to disease predictions [24], mostly using variants of RNNs with distinct network components, for example, by adding an attention mechanism to evaluate the weights of each variable [25][26][27][28][29] or by using special configurations to tackle the problem of time decays [23,25,27,[30][31][32]. Typical prediction tasks include the prediction of diabetes mellitus [23], Parkinson disease [29,33], chronic heart failure [26], sepsis [34], mortality, and readmission [25].…”