A deep reinforcement learning framework for continuous intraday market bidding

Abstract

The large integration of variable energy resources is expected to shift a large part of the energy exchanges closer to real-time, where more accurate forecasts are available. In this context, the short-term electricity markets and in particular the intraday market are considered a suitable trading floor for these exchanges to occur. A key component for the successful renewable energy sources integration is the usage of energy storage. In this paper, we propose a novel modelling framework for the strategic participation of energy storage in the European continuous intraday market where exchanges occur through a centralized order book. The goal of the storage device operator is the maximization of the profits received over the entire trading horizon, while taking into account the operational constraints of the unit. The sequential decision-making problem of trading in the intraday market is modelled as a Markov Decision Process. An asynchronous version of the fitted Q iteration algorithm is chosen for solving this problem due to its sample efficiency. The large and variable number of the existing orders in the order book motivates the use of high-level actions and an alternative state representation. Historical data are used for the generation of a large number of artificial trajectories in order to address exploration issues during the learning process. The resulting policy is back-tested and compared against a number of benchmark strategies. Finally, the impact of the storage characteristics on the total revenues collected in the intraday market is evaluated.

CIMTrade

“…DERs and intraday markets: Bouskas et. al., [10] discuss a strategy for a grid connected storage to trade in intraday markets using a deep reinforcement learning framework. They model the trading problem as a Markov Decision Process and present an asynchronous distributed version of the fitted Q iteration algorithm.…”

“…There has been some research that focuses on DERs trading in intraday markets, either exclusively or in combination with other markets [1,5,10,20,33,39]. These works aim to either reduce the cost of energy procurement or maximize the revenue generated through trading.…”

Usage of GAMS-based Digital Twins and Clustering to Improve Energetic Systems Control

“…Several approaches are being developed for energy management systems [6][7][8][9]27,28]. The latest one, based on deep reinforced learning, while producing results and being developed in open-sourced frameworks [29][30][31][32] is considered a black-box model [33] and the results are hard to explain, analyze and validate for domain experts. The approach developed in this paper can be considered as a proxy for Off-Policy reinforcement learning [34].…”

Capturing Electricity Market Dynamics in the Optimal Trading of Strategic Agents using Neural Network Constrained Optimization

“…To improve consistency between models and real-world data, reinforcement learning (RL) may be seen as an alternative to the model-based MPEC approach, leading to a fully data-driven solution method wherein the optimal policy is directly learned from interactions with the physical environment. In [7], the optimal trading strategy in the (multi-stage) intra-day market is improved using deep Q-learning, i.e., a combination of value-based RL with deep neural networks [8]. The same problem is then tackled in [9] using policybased RL, where a stochastic threshold policy is approximated with a parametric function to boost both performance and computational tractability.…”