Hyper-parameter Optimization of Multi-attention Recurrent Neural Network for Battery State-of-Charge Forecasting

Mashlakov, Aleksei; Tikka, Ville; Lensu, Lasse; Romanenko, Aleksei; Honkapuro, Samuli

doi:10.1007/978-3-030-30241-2_41

Cited by 6 publications

(6 citation statements)

References 10 publications

(11 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…MOTPE allows for the optimization of multiple performance metrics at once by analyzing the results of the current models against the currently observed frontier of solutions via using a function that calculates the expected hypervolume improvement at each step of its optimization. To the best of our knowledge, OPTUNA has not been used extensively for software engineering problems (but it has been used in domains such as disease diagnosis [77], equipment performance forecast [39] and others). We assert that it is important to baseline our new methods against OPTUNA (and MOTPE) since this is a clear extension and improvement of HYPEROPT.…”

Section: From Effort To Health Estimationmentioning

confidence: 99%

Optimizing Predictions for Very Small Data Sets: a case study on Open-Source Project Health Prediction

Lustosa¹

2023

Preprint

View full text Add to dashboard Cite

When learning from very small data sets, the resulting models can make many mistakes. For example, consider learning predictors for open source project health. The training data for this task may be very small (e.g. five years of data, collected every month means just 60 rows of training data). Using this data, prior work had unacceptably large errors in their learned predictors.We show that these high errors rates can be tamed by better configuration of the control parameters of the machine learners. For example, we present here a landscape analytics method (called SNEAK) that (a) clusters the data to find the general landscape of the hyperparameters; then (b) explores a few representatives from each part of that landscape.SNEAK is both faster and and more effective than prior state-of-the-art hyperparameter optimization algorithms (FLASH, HYPEROPT, OPTUNA, and differential evolution). For example, SNEAK found its configurations after assessing 10 2 options, while other tools needed to assess 350 to 5000 (for OPTUNA, HYPEROPT) options. More importantly, the configurations found by SNEAK had far less error that other methods. For example, for project health indicators such as 𝐶= number of commits; 𝐼 =number of closed issues, and 𝑅=number of closed pull requests, SNEAK's 12 month prediction errors are {I=0%, R=33% C=47%} while other methods have far larger errors of {I=61%,R=119% C=149%}. We conjecture that SNEAK works so well since it finds the most informative regions of the hyperparameters, then jumps to those regions. Other methods (that do not reflect over the landscape) can waste time exploring less informative options.From this, we make the following conclusions. Firstly, for predicting open source project health, we recommend landscape analytics (e.g. SNEAK). Secondly, and more generally, when learning from very small data sets, using hyperparameter optimization (e.g. SNEAK) to select learning control parameters. Due to its speed and implementation simplicity, we suggest SNEAK might also be useful in other "data-light" SE domains.To assist other researchers in repeating, improving, or even refuting our results, all our scripts and data are available on GitHub at https://github.com/zxcv123456qwe/niSneak.

show abstract

Section: From Effort To Health Estimationmentioning

confidence: 99%

Optimizing Predictions for Very Small Data Sets: a case study on Open-Source Project Health Prediction

Lustosa¹

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The learning rate is adjusted for the Adam optimization algorithm in the model. The optimization process was running for 100 trials on each of the datasets following the sequential automatic hyperparameter optimization presented in [54]. The inputs of the variational MARNN model were the hyper-parameters selected by the TPE at each trial and training and validation data of the datasets.…”

Section: Hyper-parameter Optimizationmentioning

confidence: 99%

“…Each of the trials was restricted to 20 epochs with the early stopping criterion equal to 5. More details about TPE optimizer can be found in [53] and its application to the MARNN model is described in [54].…”

Section: Hyper-parameter Optimizationmentioning

confidence: 99%

Probabilistic Forecasting of Battery Energy Storage State-of-Charge under Primary Frequency Control

Mashlakov

Lensu

Kaarna

et al. 2020

IEEE J. Select. Areas Commun.

Self Cite

View full text Add to dashboard Cite

Multi-service market optimization of battery energy storage system (BESS) requires assessing the forecasting uncertainty arising from coupled resources and processes. For the primary frequency control (PFC), which is one of the highest-value applications of BESS, this uncertainty is linked to the changes of BESS state-of-charge (SOC) under stochastic frequency variations. In order to quantify this uncertainty, this paper aims to exploit one of the recent achievements in the field of deep learning, i.e. multi-attention recurrent neural network (MARNN), for BESS SOC forecasting under PFC. Furthermore, we extend the MARNN model for probabilistic forecasting with a hybrid approach combining Mixture Density Networks and Monte Carlo dropout that incorporate the uncertainties of the data noise and the model parameters in the form of prediction interval (PI). The performance of the model is studied on BESS SOC datasets that are simulated based on real frequency measurements from three European synchronous areas in Great Britain, Continental Europe, and Northern Europe and validated by three PI evaluation indexes. Compared with the state-of-theart quantile regression algorithms, the proposed hybrid model performed well with respect to the coverage probability of PIs for the different regulatory environments of the PFC.

show abstract

“…An attention mechanism introduced in [14] is a recent advancement in the recurrent deep learning that further improves the results of vanilla RNN in memorizing long source sequences. The main difference with the RNN is that the attention develops an aggregate context vector that is filtered specifically for each output time step and memorized in the decoder layer [15]. In this study, we utilize MARNN model described in [16] that deploys lag sample values from previous input sequences at the decoding time.…”

Section: E Multi-attention Recurrent Neural Networkmentioning

confidence: 99%

Multi-Timescale Forecasting of Battery Energy Storage State-of-Charge under Frequency Containment Reserve for Normal Operation

Mashlakov

Honkapuro

Tikka

et al. 2019

2019 16th International Conference on the European Energy Market (EEM)

Self Cite

View full text Add to dashboard Cite

Forecasting the state-of-charge changes of battery energy storage, anticipated from a provision of different services, can facilitate planning of its market participation strategy and leverage the maximum potential of its energy capacity. This paper provides a performance comparison study of multiple decision-tree and data-driven machine learning methods for point forecasts of the state-of-charge of battery energy storage under frequency containment reserve for normal operation on day-, hour-, and 15-minute-ahead basis. The battery state-of-charge data for the performance evaluation were simulated with a droop curve battery model based on the historical frequency data in the northern Europe synchronous area. The results show that the data-driven methods outperform the decision-tree based methods on the 15-minute-and day-ahead time scales while demonstrating a comparable performance for the hour-ahead time scale.Index Terms-battery state-of-charge, point forecasting, decision trees, machine learning, frequency containment reserve

show abstract

Hyper-parameter Optimization of Multi-attention Recurrent Neural Network for Battery State-of-Charge Forecasting

Cited by 6 publications

References 10 publications

Optimizing Predictions for Very Small Data Sets: a case study on Open-Source Project Health Prediction

Optimizing Predictions for Very Small Data Sets: a case study on Open-Source Project Health Prediction

Probabilistic Forecasting of Battery Energy Storage State-of-Charge under Primary Frequency Control

Multi-Timescale Forecasting of Battery Energy Storage State-of-Charge under Frequency Containment Reserve for Normal Operation

Contact Info

Product

Resources

About