Policy gradient empowered LSTM with dynamic skips for irregular time series data

Weerakody, Philip B.; Wong, Kok Wai; Wang, Guanjin

doi:10.1016/j.asoc.2023.110314

Cited by 5 publications

(2 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A Long Short-Term Memory (LSTM) network is a type of Recurrent Neural Network (RNN) used for sequential data. This technique mimics the long-term and short-term memory systems in the human brain by implementing a gate system [35], that captures features and patterns within a time-series sequence [36]. Bi-directional LSTM (Bi-LSTM) considers the input sequence in both forward and backward directions, enabling longer dependency and the reversed order of features.…”

Section: Bi-lstm Modulementioning

confidence: 99%

DysDiTect: Dyslexia Identification Using CNN-Positional-LSTM-Attention Modeling with Chinese Dictation Task

Liu,

Wang,

Tong

2024

Brain Sciences

View full text Add to dashboard Cite

Handwriting difficulty is a defining feature of Chinese developmental dyslexia (DD) due to the complex structure and dense information contained within compound characters. Despite previous attempts to use deep neural network models to extract handwriting features, the temporal property of writing characters in sequential order during dictation tasks has been neglected. By combining transfer learning of convolutional neural network (CNN) and positional encoding with the temporal-sequential encoding of long short-term memory (LSTM) and attention mechanism, we trained and tested the model with handwriting images of 100,000 Chinese characters from 1064 children in Grades 2–6 (DD = 483; Typically Developing [TD] = 581). Using handwriting features only, the best model reached 83.2% accuracy, 79.2% sensitivity, 86.4% specificity, and 91.2% AUC. With grade information, the best model achieved 85.0% classification accuracy, 83.3% sensitivity, 86.4% specificity, and 89.7% AUC. These findings suggest the potential of utilizing machine learning technology to identify children at risk for dyslexia at an early age.

show abstract

Section: Bi-lstm Modulementioning

confidence: 99%

DysDiTect: Dyslexia Identification Using CNN-Positional-LSTM-Attention Modeling with Chinese Dictation Task

Liu,

Wang,

Tong

2024

Brain Sciences

View full text Add to dashboard Cite

show abstract

“…The utility of LSTM RNNs in time series analysis has gained prominence due to their capacity to encapsulate long-range dependencies and intricate temporal associations [10][11][12]. Capitalizing on this potential, this research introduces an innovative mixture attention mechanism specifically designed to elucidate the generative process of the https://doi.org/10.56578/ataiml020304 target variable.…”

Section: Introductionmentioning

confidence: 99%

Multi-Variable Time Series Decoding with Long Short-Term Memory and Mixture Attention

Seddik,

Routaib,

Elhaddadi

2023

ATAIML

View full text Add to dashboard Cite

The task of interpreting multi-variable time series data, while also forecasting outcomes accurately, is an ongoing challenge within the machine learning domain. This study presents an advanced method of utilizing Long Short-Term Memory (LSTM) recurrent neural networks in the analysis of such data, with specific attention to both target and exogenous variables. The novel approach aims to extract hidden states that are unique to individual variables, thereby capturing the distinctive dynamics inherent in multi-variable time series and allowing the elucidation of each variable's contribution to predictive outcomes. A pioneering mixture attention mechanism is introduced, which, by leveraging the aforementioned variable-specific hidden states, characterizes the generative process of the target variable. The study further enhances this methodology by formulating associated training techniques that permit concurrent learning of network parameters, variable interactions, and temporal significance with respect to the target prediction. The effectiveness of this approach is empirically validated through rigorous experimentation on three real-world datasets, including the 2022 closing prices of three major stocks -Apple (AAPL), Amazon (AMZN), and Microsoft (MSFT). The results demonstrated superior predictive performance, attributable to the successful encapsulation of the diverse dynamics of different variables. Furthermore, the study provides a comprehensive evaluation of the interpretability outcomes, both qualitatively and quantitatively. The presented framework thus holds substantial promise as a comprehensive solution that not only enhances prediction accuracy but also aids in the extraction of valuable insights from complex multi-variable datasets.

show abstract

Design of a Modified Transformer Architecture Based on Relative Position Coding

Zheng,

Gong,

Tian

et al. 2023

Int J Comput Intell Syst

View full text Add to dashboard Cite

Natural language processing (NLP) based on deep learning provides a positive performance for generative dialogue system, and the transformer model is a new boost in NLP after the advent of word vectors. In this paper, a Chinese generative dialogue system based on transformer is designed, which only uses a multi-layer transformer decoder to build the system and uses the design of an incomplete mask to realize one-way language generation. That is, questions can perceive context information in both directions, while reply sentences can only output one-way autoregressive. The above system improvements make the one-way generation of dialogue tasks more logical and reasonable, and the performance is better than the traditional dialogue system scheme. In consideration of the long-distance information weakness of absolute position coding, we put forward the improvement of relative position coding in theory, and verify it in subsequent experiments. In the transformer module, the calculation formula of self-attention is modified, and the relative position information is added to replace the absolute position coding of the position embedding layer. The performance of the modified model in BLEU, embedding average, grammatical and semantic coherence is ideal, to enhance long-distance attention.

show abstract

Policy gradient empowered LSTM with dynamic skips for irregular time series data

Cited by 5 publications

References 55 publications

DysDiTect: Dyslexia Identification Using CNN-Positional-LSTM-Attention Modeling with Chinese Dictation Task

DysDiTect: Dyslexia Identification Using CNN-Positional-LSTM-Attention Modeling with Chinese Dictation Task

Multi-Variable Time Series Decoding with Long Short-Term Memory and Mixture Attention

Design of a Modified Transformer Architecture Based on Relative Position Coding

Contact Info

Product

Resources

About