GBT: Two-stage transformer framework for non-stationary time series forecasting

Shen, Li; Wei, Yuning; Wang, Yangzhu

doi:10.1016/j.neunet.2023.06.044

Cited by 11 publications

(5 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This study introduces an innovative framework for time-series forecasting algo rithms, the Two-stage Transformer [10]. It optimizes the traditional Transformer forecast ing method's structure by segmenting it into an Auto-Regression Stage and a Self-Regres sion Stage.…”

Section: Two-stage Transformermentioning

confidence: 99%

“…However, these Transformer model variants are computationally intensive and characterized by complex parameters, leading to significant memory usage and computational resource consumption. The GBT (Two-stage Transformer) [10] addresses the critical issue of overfitting associated with improper decoder input initialization by innovatively dividing the traditional Transformer architecture into two segments: the Auto-Regression and Self-Regression stages. This approach enables the model to enhance its predictive capacity while maintaining low temporal and spatial complexity.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Predictive Model for Long-Term Lane Occupancy Rate Based on CT-Transformer and Variational Mode Decomposition

Liu,

Yu,

Liu

2024

Applied Sciences

View full text Add to dashboard Cite

Lane occupancy is a crucial indicator of traffic flow and is significant for traffic management and planning. However, predicting lane occupancy is challenging due to numerous influencing factors, such as weather, holidays, and events, which render the data nonsmooth. To enhance lane occupancy prediction accuracy, this study introduces a fusion model that combines the CT-Transformer (CSPNet-Attention and Two-stage Transformer framework) with the Temporal Convolutional Neural Network-Long Short-Term Memory (TCN-LSTM) models alongside the Variational Mode. This includes a long-term lane occupancy prediction model utilizing the Variational Mode Decomposition (VMD) technique. Initially, the Variational Mode Decomposition decomposes the original traffic flow data into multiple smooth subsequences. Subsequently, each subsequence’s autocorrelation and partial correlation coefficients ascertain the presence of seasonal characteristics. Based on these characteristics, the CT-Transformer and TCN-LSTM models process each subsequence for long-term lane occupancy rate prediction, respectively. Finally, predictions from both models are integrated using variable modes to derive the ultimate lane occupancy predictions. The core CT-Transformer model, an enhancement of the GBT (Two-stage Transformer) model, comprises two phases: autoregressive and prediction. The autoregressive phase leverages historical data for initial predictions inputted into the prediction phase. Here, the novel CSPNet-Attention mechanism replaces the conventional attention mechanism in the Encoder, reducing memory usage and computational resource loss, thereby enhancing the model’s accuracy and robustness. Experiments on the PeMS public dataset demonstrate that the proposed model surpasses existing methods in predicting long-term lane occupancy, offering decent reliability and generalizability.

show abstract

Section: Two-stage Transformermentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Predictive Model for Long-Term Lane Occupancy Rate Based on CT-Transformer and Variational Mode Decomposition

Liu,

Yu,

Liu

2024

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…To mitigate the decline in predictive performance caused by non-stationary data, classical statistical methods such as ARMA and ARIMA [20] can achieve stationarity of time series through difference operations. However, these traditional models cannot compete with deep-learning-based models in capturing dynamic changes in data and solving long-sequence time series forecasting problems [21]. With deep learning models, some researchers have attempted to alleviate the non-stationarity in the original time series by normalizing the global data distribution of related time series into a specific distribution with mean 0 and variance 1, commonly referred to as Z-score standardization [15,22].…”

Section: Introductionmentioning

confidence: 99%

“…To address the issue of non-stationarity in deep learning models due to differences in the statistical properties of input windows, Shen proposed a two-level transformer framework, which exhibited excellent stability [21]. On the other hand, Liu effectively mitigated the overfitting phenomenon caused by non-stationary data such as weather and electricity consumption predictions by optimizing the attention mechanism in the transformer model [24].…”

Section: Introductionmentioning

confidence: 99%

Study on Prediction of Zinc Grade by Transformer Model with De-Stationary Mechanism

Peng,

Luo,

Luo

et al. 2024

Minerals

View full text Add to dashboard Cite

At present, in the mineral flotation process, flotation data are easily influenced by various factors, resulting in non-stationary time series data, which lead to overfitting of prediction models, ultimately severely affecting the accuracy of grade prediction. Thus, this study proposes a de-stationary attention mechanism based on the transformer model (DST) to learn non-stationary information in raw mineral data sequences. First, normalization processing is performed on matched flotation data and mineral grade values, to make the data sequences stationary, thereby enhancing model prediction capabilities. Then, the proposed de-stationary attention mechanism is employed to learn the temporal dependencies of mineral flotation data in the transformed vanilla transformer model, i.e., non-stationary information in the mineral data sequences. Lastly, de-normalization processing is conducted to maintain the mineral prediction results within the same scale as the original data. Compared with existing models such as RNN, LSTM, transformer, Enc-Dec (RNN), and STS-D, the DST model reduced the RMSE by 20.8%, 20.8%, 62.8%, 20.5%, and 49.1%, respectively.

show abstract

“…There is an effort to exploit the emergent advances in deep-learning machine models (such as Transformers and GANs) for time series analysis [11,12]. Despite the great success achieved by these models in processing natural languages and generating images and videos, Long short-term memory (LSTM) and gated recurrent unit (GRU) networks still maintain their popularity for time series forecasting [13,14,15]. Their popularity could be attributed to their ability to effectively capture temporal dependencies, which is crucial for accurately predicting future values in a time series.…”

Section: Introductionmentioning

confidence: 99%

Enhancing Multi-Step Brent Oil Price Forecasting with Ensemble Multi-Scenario Bi-GRU Networks

Alruqimi,

Persio

2024

Preprint

View full text Add to dashboard Cite

Despite numerous research efforts in applying deep learning to time series forecasting, achieving high accuracy in multi-step predictions for volatile time series like crude oil prices remains a significant challenge. Moreover, most existing approaches primarily focus on one-step forecasting, and the performance often varies depending on the dataset and specific case study. In this paper, we introduce an ensemble model to capture Brent oil price volatility and enhance the multi-step prediction. Our methodology employs a two-pronged approach. First, we assess popular deep-learning models and the impact of various external factors on forecasting accuracy. Then, we introduce an ensemble multi-step forecasting model for Brent oil prices. Our approach generates accurate forecasts by employing ensemble techniques across multiple forecasting scenarios using three BI-GRU networks. Extensive experiments were conducted on a dataset encompassing the COVID-19 pandemic period, which had a significant impact on energy markets. The proposed model's performance was evaluated using the standard evaluation metrics of MAE, MSE, and RMSE. The results demonstrate that the proposed model outperforms benchmark and established models.

show abstract

GBT: Two-stage transformer framework for non-stationary time series forecasting

Cited by 11 publications

References 11 publications

Predictive Model for Long-Term Lane Occupancy Rate Based on CT-Transformer and Variational Mode Decomposition

Predictive Model for Long-Term Lane Occupancy Rate Based on CT-Transformer and Variational Mode Decomposition

Study on Prediction of Zinc Grade by Transformer Model with De-Stationary Mechanism

Enhancing Multi-Step Brent Oil Price Forecasting with Ensemble Multi-Scenario Bi-GRU Networks

Contact Info

Product

Resources

About