Network self attention for forecasting time series

Hu, Yuntong; Xiao, Fuyuan

doi:10.1016/j.asoc.2022.109092

Cited by 29 publications

(6 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Self-Attention mechanism helped extract important information by changing the weights of different channels, addressing the significance of different channels and samples when feature extraction is involved. YuntongHu et al [24] proposed a new time series prediction model that learns similarity scores based on network Self-Attention. This approach improved prediction accuracy and robustness.…”

Section: Proposed Methods a Mgts-attentionmentioning

confidence: 99%

Load Energy Decomposition Algorithm Based on Improved Bidirectional Transformer Combined With Time-Sensing Self-Attention

Xuan,

Pang,

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Non-Intrusive Load Monitoring (NILM) is a technique capable of extracting the quantity, type and operational status from overall power data in a specific area. Through energy disaggregation, the total power consumption can be decomposed into device-level power consumption. In recent years, with the advancement of deep learning techniques, numerous improvements have been made in energy disaggregation technologies. However, for NILM tasks, significant variations in device switching can result in data imbalance, and devices in different operating states can impact the model's prediction accuracy. To address these issues, this paper proposes a method based on Bidirectional Transformer and GRU, combined with time-aware self-attention for load prediction(CTA-BERT). In this model, We leverage the bidirectional transformer of BERT to learn features at different positions in the power sequence. Simultaneously, GRU is added to capture long-term sequence dependencies. Device time-state variables and dilated convolutional layers are introduced in the Transformer section to effectively capture long sequence dependencies and global features. We also enhance the masking strategy to adapt the appliances in different states. Finally, on the REDD and UK-DALE datasets, we tested and compared the proposed method with several state-ofthe-art NILM algorithms. The results indicate that the model's average absolute error (MAE) is reduced by an average of 25% across all devices, F1-Score is improved by approximately 20%, and prediction accuracy is significantly enhanced.

show abstract

Section: Proposed Methods a Mgts-attentionmentioning

confidence: 99%

Load Energy Decomposition Algorithm Based on Improved Bidirectional Transformer Combined With Time-Sensing Self-Attention

Xuan,

Pang,

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Inspired by the human brain's attention mechanism, the attention mechanism selectively weighs critical information from a large amount of data to enhance neural network efficiency. This mechanism has found extensive applications in various domains, including sentiment analysis [22,23], image segmentation [24,25], intelligent recommendation [26,27] and time series forecasting [28][29][30]. The self-attention mechanism, a prominent component of the transformer model developed by the Google team [31], is a variant of the attention mechanism.…”

Section: The Attention Mechanismmentioning

confidence: 99%

Enhancing Anomaly Detection for Cultural Heritage via Long Short-Term Memory with Attention Mechanism

Wu,

Dong,

Shan

et al. 2024

Electronics

View full text Add to dashboard Cite

Cultural heritages are invaluable and non-renewable resources. Existing warning mechanisms usually rely on degradation analysis to assess environmental risk factors. However, they have limitations such as complex research, poor generalization, and inadequate warnings. To address these issues, we propose a hybrid model that combines the long short-term memory network (LSTM) and attention mechanisms with environmental factors to detect anomalies in cultural heritage. The attention mechanism extracts temporal dependencies, while LSTM captures global long-term patterns, enhancing the reconstruction capability. Additionally, we introduce the seasonal, trend, shapelet, and mixed anomalies to enrich the positive and negative samples and propose a novel threshold extraction method to reduce the reliance on prior knowledge. Extensive experiments demonstrate that LSTM-Attention outperforms previous methods, with a root mean square error (RMSE) of 34.328, mean absolute error (MAE) of 27.060, and the largest area under the receiver operating characteristic curve (AUC) value, highlighting the superiority and effectiveness of our method. The ablation study proves the contribution of the attention mechanism, and the reconstruction step can be flexibly adjusted according to practical needs. Finally, the algorithm has been successfully implemented in engineering and employed in practice, providing valuable guidelines for the preservation of ancient buildings worldwide.

show abstract

“…MHA is based on the self-attention mechanism (self attention) [40], which is a method for obtaining contextual information by calculating the correlations between different positions in an input sequence. Since there are many different forms of correlation and many different definitions, it is sometimes not possible to focus on just one form of correlation, but rather on multiple forms of correlation.…”

Section: Multihead Attentionmentioning

confidence: 99%

A Novel Dual-Channel Temporal Convolutional Network for Photovoltaic Power Forecasting

Ren,

Zhang,

Sun

et al. 2024

Energies

View full text Add to dashboard Cite

A large proportion of photovoltaic (PV) power generation is connected to the power grid, and its volatility and stochasticity have significant impacts on the power system. Accurate PV power forecasting is of great significance in optimizing the safe operation of the power grid and power market transactions. In this paper, a novel dual-channel PV power forecasting method based on a temporal convolutional network (TCN) is proposed. The method deeply integrates the PV station feature data with the model computing mechanism through the dual-channel model architecture; utilizes the combination of multihead attention (MHA) and TCN to extract the multidimensional spatio-temporal features between other meteorological variables and the PV power; and utilizes a single TCN to fully extract the temporal constraints of the power sequence elements. The weighted fusion of the dual-channel feature data ultimately yields the ideal forecasting results. The experimental data in this study are from a 26.52 kW PV power plant in central Australia. The experiments were carried out over seven different input window widths, and the two models that currently show superior performance within the field of PV power forecasting: the convolutional neural network (CNN), and the convolutional neural network combined with a long and short-term memory network (CNN_LSTM), are used as the baseline models. The experimental results show that the proposed model and the baseline models both obtained the best forecasting performance over a 1-day input window width, while the proposed model exhibited superior forecasting performance compared to the baseline model. It also shows that designing model architectures that deeply integrate the data input method with the model mechanism has research potential in the field of PV power forecasting.

show abstract

Network self attention for forecasting time series

Cited by 29 publications

References 27 publications

Load Energy Decomposition Algorithm Based on Improved Bidirectional Transformer Combined With Time-Sensing Self-Attention

Load Energy Decomposition Algorithm Based on Improved Bidirectional Transformer Combined With Time-Sensing Self-Attention

Enhancing Anomaly Detection for Cultural Heritage via Long Short-Term Memory with Attention Mechanism

A Novel Dual-Channel Temporal Convolutional Network for Photovoltaic Power Forecasting

Contact Info

Product

Resources

About