Are Transformers Effective for Time Series Forecasting?

Abstract

Recently, there has been a surge of Transformer-based solutions for the time series forecasting (TSF) task, especially for the challenging long-term TSF problem. Transformer architecture relies on self-attention mechanisms to effectively extract the semantic correlations between paired elements in a long sequence, which is permutation-invariant and "anti-ordering" to some extent. However, in time series modeling, we are to extract the temporal relations among an ordering set of continuous points. Consequently, whether Transformer-based techniques are the right solutions for "long-term" time series forecasting is an interesting problem to investigate, despite the performance improvements shown in these studies. In this work, we question the validity of Transformer-based TSF solutions. In their experiments, the compared (non-Transformer) baselines are mainly autoregressive forecasting solutions, which usually have a poor long-term prediction capability due to inevitable error accumulation effects. In contrast, we use an embarrassingly simple architecture named DLinear that conducts direct multi-step (DMS) forecasting for comparison. DLinear decomposes the time series into a trend and a remainder series and employs two one-layer linear networks to model these two series for the forecasting task. Surprisingly, it outperforms existing complex Transformer-based models in most cases by a large margin. Therefore, we conclude that the relatively higher long-term forecasting accuracy of Transformer-based TSF solutions shown in existing works has little to do with the temporal relation extraction capabilities of the Transformer architecture. Instead, it is mainly due to the non-autoregressive DMS forecasting strategy used in them. We hope this study also advocates revisiting the validity of Transformer-based solutions for other time series analysis tasks (e.g., anomaly detection) in the future. Code is available at https://github.com/cure-lab/DLinear. * Equal contribution Preprint. Under review.

A Digitalization Framework for Smart Maintenance of Historic Buildings

“…In addition to linear regression (LR), seven deep learning methods, namely hierarchical interpolation for time series forecasting (N-HiTS) [66], temporal convolutional network (TCN) [67], Transformer (TF) [68], NLinear [69], long short-term memory (LSTM) [70], gated recurrent unit (GRU) [71], and temporal fusion transformer (TFT) [53], were investigated to develop predictive models and compare their performance. N-HiTS and NLinear were improved to support producing probabilistic forecasts based on quantile regression.…”

State Causality and Adaptive Covariance Decomposition Based Time Series Forecasting

“…These models show that sequence dependencies can be modeled for long-term forecasting [ 27 ], but rely heavily on periodicity. In general, our findings suggest that the correlation between subsequences is crucial.…”

MLGN: multi-scale local-global feature learning network for long-term series forecasting

“…Both modules are used to replace self-attention and cross-attention modules, with a time complexity of O (L). These Transformer-based models have demonstrated outstanding performance in learning long-term sequence dependencies [38]. CNN-based methods are commonly used to extract local temporal features by leveraging convolution kernels.…”

“…The overall structure of MLGN is shown in figure 2. Inspired by traditional time series decomposition algorithms [46] and deep learning models [24,30,37,38], we design a multi-scale sequence decomposition (MSDecomp) block to separate complex patterns of input series. Then we utilize seasonal component prediction module to predict seasonal information and trend component prediction module to predict trend information respectively.…”