Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting

Abstract

Many real-world applications require the prediction of long sequence time-series, such as electricity consumption planning. Long sequence time-series forecasting (LSTF) demands a high prediction capacity of the model, which is the ability to capture precise long-range dependency coupling between output and input efficiently. Recent studies have shown the potential of Transformer to increase the prediction capacity. However, there are several severe issues with Transformer that prevent it from being directly applicable to LSTF, including quadratic time complexity, high memory usage, and inherent limitation of the encoder-decoder architecture. To address these issues, we design an efficient transformer-based model for LSTF, named Informer, with three distinctive characteristics: (i) a ProbSparse self-attention mechanism, which achieves O(L log L) in time complexity and memory usage, and has comparable performance on sequences' dependency alignment. (ii) the self-attention distilling highlights dominating attention by halving cascading layer input, and efficiently handles extreme long input sequences. (iii) the generative style decoder, while conceptually simple, predicts the long time-series sequences at one forward operation rather than a step-by-step way, which drastically improves the inference speed of long-sequence predictions. Extensive experiments on four large-scale datasets demonstrate that Informer significantly outperforms existing methods and provides a new solution to the LSTF problem.

Grasping Core Rules of Time Series through Pure Models

“…The following development in the time series was caused by the attention mechanism. With the help of Informer (Zhou et al 2021), DA-RNN (Qin et al 2017), the precision of the attention mechanism was improved. Graph neural networks are new models that convert dense connections to sparse graph structures, fitting information in explicit or implicitly graph-structured data, such as MTGNN (Wu et al 2020).…”

“…In this chapter, we will analyze two datasets with contrasting characteristics and their extension tasks, validate the predictive capability of PureTS, and compare the running times and parameter sizes of several models horizontally. The eight publicly available benchmark datasets are split into short sequence prediction and long sequence prediction in accordance with the previous experimental settings (Zhang et al 2022;Zhou et al 2021;Lai et al 2018) Reformer (Kitaev, Kaiser, and Levskaya 2020). Informer (Zhou et al 2021).…”

“…The eight publicly available benchmark datasets are split into short sequence prediction and long sequence prediction in accordance with the previous experimental settings (Zhang et al 2022;Zhou et al 2021;Lai et al 2018) Reformer (Kitaev, Kaiser, and Levskaya 2020). Informer (Zhou et al 2021). Autoformer (Wu et al 2021).…”

Mining Reaction and Diffusion Dynamics in Social Activities

“…In particular, time series forecasting models using Transformer, developed in the field of a natural language preocessing [54], achieve high prediction performance [12,31,33]. For example, Informer [58], one of the Transformer-based models, achieves a successful outcome in long-ahead forecasting. Deep learning-based models guarantee high forecasting performance; however, they are still problematic in terms of their interpretability.…”

“…• Informer [58], which is a transformer-based model based on ProbSparse self-attention and self-attention distilling, is known for its remarkable efficiency in long time series forecasting. We select a two-layer stack for both encoder and decoder and set 26 (half of the prediction sequence length) as the token length of the decoder.…”