Evaluation of the Transformer Architecture for Univariate Time Series Forecasting

Lara-Benítez, Pedro; Gallego-Ledesma, Luis; Carranza-García, Manuel; Luna-Romera, José María

doi:10.1007/978-3-030-85713-4_11

Cited by 10 publications

(4 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A Decoder-only Transformer structure-based method, geared towards univariate time-series forecasting, is introduced (Lara- Benítez et al, 2021). In this investigation, the conventional Transformer Decoder blocks were adhered to, wherein each decoder block comprises a masked self-attention module followed by multihead attention and a feed-forward block.…”

Section: Related Workmentioning

confidence: 99%

IoT-Based Energy Consumption Prediction Using Transformers

Alıoghlı,

Yıldırım Okay

2024

Gazi University Journal of Science Part A: Engineering and Innovation

View full text Add to dashboard Cite

With the advancement of various IoT-based systems, the amount of data is steadily increasing. The increase of data on a daily basis is essential for decision-makers to assess current situations and formulate future policies. Among the various types of data, time-series data presents a challenging relationship between current and future dependencies. Time-series prediction aims to forecast future values of target variables by leveraging insights gained from past data points. Recent advancements in deep learning-based algorithms have surpassed traditional machine learning-based algorithms for time-series in IoT systems. In this study, we employ Enc&Dec Transformer, the latest advancements in neural networks for time-series prediction problems. The obtained results were compared with Encoder-only and Decoder-only Transformer blocks as well as well-known recurrent based algorithms, including 1D-CNN, RNN, LSTM, and GRU. To validate our approach, we utilize three different univariate time-series datasets collected on an hourly basis, focusing on energy consumption within IoT systems. Our results demonstrate that our proposed Transformer model outperforms its counterparts, achieving a minimum Mean Squared Error (MSE) of 0.020 on small, 0.008 on medium, and 0.006 on large-sized datasets.

show abstract

Section: Related Workmentioning

confidence: 99%

IoT-Based Energy Consumption Prediction Using Transformers

Alıoghlı,

Yıldırım Okay

2024

Gazi University Journal of Science Part A: Engineering and Innovation

View full text Add to dashboard Cite

show abstract

“…The seq2seq [41] model based on the Attention mechanism has emerged in prediction [42], but it has not yet been used in univariate time series prediction. A study [43] comparing Transformer and LSTM solutions to prediction problems pointed out the limitations of Transformer in terms of computation and parameter handling.…”

Section: Related Work a Univariate Time Series Forecastingmentioning

confidence: 99%

Umformer: A Transformer Dedicated to Univariate Multistep Prediction

Chen

et al. 2022

IEEE Access

View full text Add to dashboard Cite

Univariate multi-step time series forecasting (UMTF) has many applications, such as the forecast of access traffic. The solution to the UMTF problem needs to efficiently capture key information in univariate data and improve the accuracy of multi-step forecasting. The advent of deep learning (DL) enables multi-level, high-performance prediction of complex multivariate inputs, but the solution and research of UMTF problems is extremely scarce. Existing methods cannot satisfy recent univariate forecasting tasks in terms of forecasting accuracy, efficiency, etc. This paper proposes a Transformer-based univariate multistep forecasting model: Umformer. The contributions include: (1) To maximize the information obtained from a single variable, we propose a Prophet-based method for variable extraction, additionally considering some correlated variables for accurate predictions. (2) Gated linear units variants with three weight matrices (GLUV3) are designed, as a gating to improve the function of selective memory in long sequences, thereby obtaining more helpful information from a limited number of univariate variables and improving prediction accuracy. (3) Shared Double-heads Probsparse Attention (SDHPA) mechanism reduces memory footprint and improves attention-awareness. We combine the latest research results of current DL technology to achieve high-precision prediction in UMTF.Extensive experiments on public datasets from five different domains have shown that five metrics demonstrate that the Umformer approach is significantly better than existing methods. We offer a more efficient solution for UMTF.

show abstract

“…Unlike LSTM, CNN, and GRU, which focus more on sequence-based learning, the parallel approach of Transformer allows it to overcome long-range constraints in time series. Recent studies [14]- [16] have demonstrated that Deep Transformer (DT) models with attention mechanisms outperform LSTM, CNN, and GRU in time series prediction tasks.…”

mentioning

confidence: 99%

AutoCyclic: Deep Learning Optimizer for Time Series Data Prediction

Arthur,

Yudistira,

Dewi

2024

IEEE Access

View full text Add to dashboard Cite

Time series prediction, a pivotal component in various domains, faces the intricate task of capturing complex patterns and dependencies within sequential data. Autocyclic Learning Rate (Auto-Cyclic) emerges as a groundbreaking solution, seamlessly integrating cosine cyclic learning rates with a meticulous emphasis on autocorrelation and variance. In the realm of time series forecasting, where longterm dependencies and dynamic patterns pose significant challenges, AutoCyclic represents a transformative approach. By dynamically adjusting learning rates based on the nuanced characteristics of time series data, AutoCyclic not only addresses issues such as underfitting and overfitting, but also provides exceptional robustness to outliers. Through rigorous evaluations on diverse datasets, including ETTm2, M4, and WindTurbine, AutoCyclic consistently outperforms traditional optimizers such as Adams Optimizer and Cosine Cyclic Learning Rate, showcasing the lowest Mean Absolute Percentage Error (MAPE) and Mean Absolute Error (MAE) values. What sets AutoCyclic apart is not only its predictive accuracy, but also its astute exploitation of autocorrelation, which provides a unique advantage in discerning and leveraging inherent patterns in time series data. This intrinsic capability positions AutoCyclic as a transformative tool, redefining the landscape of time series forecasting methodologies and opening new dimensions in the utilization of autocorrelation for more insightful and accurate predictions.

show abstract

Evaluation of the Transformer Architecture for Univariate Time Series Forecasting

Cited by 10 publications

References 16 publications

IoT-Based Energy Consumption Prediction Using Transformers

IoT-Based Energy Consumption Prediction Using Transformers

Umformer: A Transformer Dedicated to Univariate Multistep Prediction

AutoCyclic: Deep Learning Optimizer for Time Series Data Prediction

Contact Info

Product

Resources

About