Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting

Abstract

Many real-world applications require the prediction of long sequence time-series, such as electricity consumption planning. Long sequence time-series forecasting (LSTF) demands a high prediction capacity of the model, which is the ability to capture precise long-range dependency coupling between output and input efficiently. Recent studies have shown the potential of Transformer to increase the prediction capacity. However, there are several severe issues with Transformer that prevent it from being directly applicable to LSTF, such as quadratic time complexity, high memory usage, and inherent limitation of the encoder-decoder architecture. To address these issues, we design an efficient transformer-based model for LSTF, named Informer, with three distinctive characteristics: (i) a ProbSparse Self-attention mechanism, which achieves O(L log L) in time complexity and memory usage, and has comparable performance on sequences' dependency alignment. (ii) the self-attention distilling highlights dominating attention by halving cascading layer input, and efficiently handles extreme long input sequences. (iii) the generative style decoder, while conceptually simple, predicts the long time-series sequences at one forward operation rather than a step-by-step way, which drastically improves the inference speed of long-sequence predictions. Extensive experiments on four large-scale datasets demonstrate that Informer significantly outperforms existing methods and provides a new solution to the LSTF problem.

DAE-PINN: A Physics-Informed Neural Network Model for Simulating Differential-Algebraic Equations with Application to Power Networks

“…On using more sophisticated Neural Network architectures: In this paper, to simulate DAEs over a long-time horizon, we employed a modified version of the conventional fully −0.5 0.0 0.5 connected neural network architecture. However, we realize that other architectures may increase our ability to simulate long-time dependencies [23]. Thus, in our future work, we plan to implement DAE-PINN using neural networks that can generalize well to unseen events.…”

End-to-End User Behavior Retrieval in Click-Through RatePrediction Model

“…Besides the above related works on CTR prediction task, plenty of works aim to improve the efficiency and effectiveness of transformer. Reformer [13] and Informer [37] are the most relevant Each 𝑑-dimensional vector is converted to a 𝑚-length signature vector. Each random rotation here can be regarded as one "hash function".…”

Transformer-Encoder-GRU (T-E-GRU) for Chinese Sentiment Analysis on Chinese Comment Text

“…Vaswani et al completely abandoned RNN and CNN to build transformer which is entirely based on the fully-connected layer and attention mechanism [35]. In 2020, Zhou H et al proposed probsparse self-attention mechanism and informer, and this model achieved high prediction capacity in the long sequence time-series forecasting [41]. Transformer is divided into two parts: encoding and decoding.…”