Residential Demand Response of Thermostatically Controlled Loads Using Batch Reinforcement Learning

Abstract

Driven by recent advances in batch Reinforcement Learning (RL), this paper contributes to the application of batch RL to demand response. In contrast to conventional modelbased approaches, batch RL techniques do not require a system identification step, making them more suitable for a large-scale implementation. This paper extends fitted Q-iteration, a standard batch RL technique, to the situation when a forecast of the exogenous data is provided. In general, batch RL techniques do not rely on expert knowledge about the system dynamics or the solution. However, if some expert knowledge is provided, it can be incorporated by using the proposed policy adjustment method. Finally, we tackle the challenge of finding an open-loop schedule required to participate in the day-ahead market. We propose a model-free Monte Carlo method that uses a metric based on the state-action value function or Q-function and we illustrate this method by finding the day-ahead schedule of a heat-pump thermostat. Our experiments show that batch RL techniques provide a valuable alternative to model-based controllers and that they can be used to construct both closed-loop and open-loop policies.

Reinforcement Learning for Electric Power System Decision and Control: Past Considerations and Perspectives

“…The usual roles of domain knowledge are: · Making the computations necessary for solving the problem more time-or space-efficient, · Guiding the solution process, ( (2009)). Work presented in Ruelens et al (2016) extended the fitted Q iteration algorithm in order to take advantage of domain-specific knowledge (in particular case a forecast of the exogenous data is provided to design demand response control). Q(λ) with eligibility traces is used to take advantage of domain-specific knowledge in Yu et al (2011).…”

“…Power system components considered include: dynamic brake Ernst et al (2004); Glavic (2005), thyristor controlled series capacitor Ernst et al (2004Ernst et al ( , 2009, quadrature booster Li and Wu (1999), synchronous generators (all AGC related references), individual or aggregated loads Vandael et al (2015); Ruelens et al (2016), etc. If used as a multi-agent system, then additional state variables must be introduced to ensure convergence of these essentially distributed computation schemes, and an adapted variant of standard RL methods is often used (for example correlated equilibrium Q(λ) Yu et al (2012a)).…”

Battery Energy Management in a Microgrid Using Batch Reinforcement Learning

“…This results in a slow convergence rate of the Q-learning algorithm to an optimal policy [21]; more observations are needed to construct a control policy. In batch RL techniques (off-line RL) [22,23], a controller estimates a control policy based on a batch of its past experiences.…”

“…The ability of batch RL to reuse their past experiences makes them converge faster than online RL methods like Q-learning and SARSA. Batch RL has been used for demand response in [21,[24][25][26]. Vandael et al [27] used a batch RL technique to find a day-ahead consumption plan of a cluster of electric vehicles.…”

Aggregation of power capabilities of heterogeneous resources for real-time control of power grids

“…The behaviour of these representative devices allows to capture the behaviour of the entire set of the aggregated ones (e.g., [1], [2], [3], [4]). The second approach is model-free since it infers the behaviour of the distributed devices from the interaction between them and a central unit (i.e., the aggregator) (e.g., [5], [6]). Usually, these approaches adopt data-driven learning techniques.…”