Convolutional Long-Short Term Memory Network with Multi-Head Attention Mechanism for Traffic Flow Prediction

Wei, Yupeng; Liu, Hongrui

doi:10.3390/s22207994

Cited by 12 publications

(7 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The decoder LSTM takes the encoding and generates the forecast,

, where M is the number of steps to forecast. The decoder LSTM also has an attention mechanism which allows it to focus on different parts of the encoding at each time step and generate the forecast one step at a time [ 47 ]. At time step t , the decoder computes the attention weights

over the encoding and calculates the context vector

as a weighted sum of the encodings, as follows:

…”

Section: Methodsmentioning

confidence: 99%

Optimized EWT-Seq2Seq-LSTM with Attention Mechanism to Insulators Fault Prediction

Klaar

Stefenon

Seman

et al. 2023

Sensors

View full text Add to dashboard Cite

Insulators installed outdoors are vulnerable to the accumulation of contaminants on their surface, which raise their conductivity and increase leakage current until a flashover occurs. To improve the reliability of the electrical power system, it is possible to evaluate the development of the fault in relation to the increase in leakage current and thus predict whether a shutdown may occur. This paper proposes the use of empirical wavelet transform (EWT) to reduce the influence of non-representative variations and combines the attention mechanism with a long short-term memory (LSTM) recurrent network for prediction. The Optuna framework has been applied for hyperparameter optimization, resulting in a method called optimized EWT-Seq2Seq-LSTM with attention. The proposed model had a 10.17% lower mean square error (MSE) than the standard LSTM and a 5.36% lower MSE than the model without optimization, showing that the attention mechanism and hyperparameter optimization is a promising strategy.

show abstract

“…The decoder LSTM takes the encoding and generates the forecast,

over the encoding and calculates the context vector

as a weighted sum of the encodings, as follows:

…”

Section: Methodsmentioning

confidence: 99%

Optimized EWT-Seq2Seq-LSTM with Attention Mechanism to Insulators Fault Prediction

Klaar

Stefenon

Seman

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

“…In order to make the constructed predictor better explore the influence mechanism of yield of molten iron and improve the prediction accuracy of the neural network, this article stacks multiple DAEs to build a deep neural network, so that the high-level hidden layer can learn more robust feature analysis capabilities, the process of which is shown in Figure 5. The attention mechanism can deeply explore the long-distance relationship between labels through the global interaction analysis of the data, which has advantages in the extraction of highdimensional features, is generally divided into self-attention mechanism 23) and multi-head attention mechanism 24) . Self-attention mechanism concentrates attention on the current location without perceiving other locations during the encoding process.…”

Section: 𝑓 𝑙mentioning

confidence: 99%

Dynamic Prediction Model of Yield of Molten Iron Based on Multi-Head Attention Mechanism

Duan,

Liu,

et al. 2024

ISIJ Int.

View full text Add to dashboard Cite

：Efficient matching of the preheat-numbers of the molten iron ladle with yield of molten iron in a single iron time is conducive to reducing energy waste, so it is of great economic significance to iron and steel enterprises to accurately predict yield of molten iron according to the characteristics of the per furnace. In this article, considering the variation of yield of molten iron and corresponding synergistic parameters depending on furnace conditions, we use the multi-head attention mechanism to capture the Attention-score of the feature parameters, and the weight matrix of the predictor is dynamically adjusted, so that it can assign more training weights to high-value feature parameters in real time according to the change of furnace conditions, efficiently complete the prediction of yield of molten iron. First, twenty characteristics are selected by mutual information method. Then, the influence degree of selected features on yield of molten iron is calculated in real time. Finally, stacked denoising auto encoder (SDAE) is used to train the deep neural network, and we constructed the predictor after adjusting the parameters to optimal. The test results show that the prediction accuracy of proposed model is 95% under the error range of ±50t, which is higher than traditional SVM, DAE, and SDAE model without the multi-head attention mechanism. Ultimately, we analyze and quantify the influence process of 20 factors on yield of molten iron, and develop a dynamic prediction system of yield of molten iron based on proposed model, which has good applicability for the prediction of yield of molten iron and effectively guides the transfer scheduling of molten iron ladles.Keywords：Dynamic prediction model of yield of molten iron; Multi-head attention mechanism; Stacked denoising auto encoder; Transfer scheduling of molten iron ladles;

show abstract

“…Wei et al [ 26 ] proposed a model called AutoEncoder Long Short-Term Memory (AE-LSTM), which uses AutoEncoder to capture the internal relationship of the traffic flow by extracting the characteristics of upstream and downstream traffic flow data and employs LSTM to predict the complex linear traffic flow data. Wei et al [ 27 ] proposed a decoder convolutional LSTM model, where the convolutional operation is used to consider the correlation of the high-dimensional features, and the LSTM network is used to consider the temporal correlation of traffic flow data. Moreover, the multi-head attention mechanism is introduced to use the most relevant portion of the traffic data to improve the prediction performance.…”

Section: Related Workmentioning

confidence: 99%

Incorporating Multivariate Auxiliary Information for Traffic Prediction on Highways

Xiong

Wan

et al. 2023

Sensors

View full text Add to dashboard Cite

Traffic flow prediction is one of the most important tasks of the Intelligent Transportation Systems (ITSs) for traffic management, and it is also a challenging task affected by many complex factors, such as weather and time. Many cities adopt efficient traffic prediction methods to control traffic congestion. However, most of the existing methods of traffic prediction focus on urban road scenarios, neglecting the complexity of multivariate auxiliary information in highways. Moreover, these methods have difficulty explaining the prediction results based only on the historical traffic flow sequence. To tackle these problems, we propose a novel traffic prediction model, namely Multi-variate and Multi-horizon prediction based on Long Short-Term Memory (MMLSTM). MMLSTM can effectively incorporate auxiliary information, such as weather and time, based on a strategy of multi-horizon time spans to improve the prediction performance. Specifically, we first exploit a multi-horizon bidirectional LSTM model for fusing the multivariate auxiliary information in different time spans. Then, we combine an attention mechanism and multi-layer perceptron to conduct the traffic prediction. Furthermore, we can use the information of multivariate (weather and time) to provide interpretability to manage the model. Comprehensive experiments are conducted on Hangst and Metr-la datasets, and MMLSTM achieves better performance than baselines on traffic prediction tasks.

show abstract

Convolutional Long-Short Term Memory Network with Multi-Head Attention Mechanism for Traffic Flow Prediction

Cited by 12 publications

References 46 publications

Optimized EWT-Seq2Seq-LSTM with Attention Mechanism to Insulators Fault Prediction

Optimized EWT-Seq2Seq-LSTM with Attention Mechanism to Insulators Fault Prediction

Dynamic Prediction Model of Yield of Molten Iron Based on Multi-Head Attention Mechanism

Incorporating Multivariate Auxiliary Information for Traffic Prediction on Highways

Contact Info

Product

Resources

About