Data-Driven Online Energy Scheduling of a Microgrid Based on Deep Reinforcement Learning

Abstract

The proliferation of distributed renewable energy resources (RESs) poses major challenges to the operation of microgrids due to uncertainty. Traditional online scheduling approaches relying on accurate forecasts become difficult to implement due to the increase of uncertain RESs. Although several data-driven methods have been proposed recently to overcome the challenge, they generally suffer from a scalability issue due to the limited ability to optimize high-dimensional continuous control variables. To address these issues, we propose a data-driven online scheduling method for microgrid energy optimization based on continuous-control deep reinforcement learning (DRL). We formulate the online scheduling problem as a Markov decision process (MDP). The objective is to minimize the operating cost of the microgrid considering the uncertainty of RESs generation, load demand, and electricity prices. To learn the optimal scheduling strategy, a Gated Recurrent Unit (GRU)-based network is designed to extract temporal features of uncertainty and generate the optimal scheduling decisions in an end-to-end manner. To optimize the policy with high-dimensional and continuous actions, proximal policy optimization (PPO) is employed to train the neural network-based policy in a data-driven fashion. The proposed method does not require any forecasting information on the uncertainty or a prior knowledge of the physical model of the microgrid. Simulation results using realistic power system data of California Independent System Operator (CAISO) demonstrate the effectiveness of the proposed method.

Optimization Models under Uncertainty in Distributed Generation Systems: A Review

“…QP/CP Solver Simulation Platform [163] CPLEX GAMS [70,144] MATLAB [63] CVX MATLAB [168] Active set solver MATLAB [51,91,172] Gurobi MATLAB [81] Python [60] MOSEK Python [29,107,109,127] Other/Unspecified Unspecified…”

Minimizing Energy Cost in PV Battery Storage Systems Using Reinforcement Learning

“…However, because the Q-learning algorithm only allows discrete states and actions, a discretization must be performed in each of these papers. As described in Ji et al [26], the discretization of actions can degrade the performance of the EM and becomes unfeasible with higher dimensionality of the action space. Therefore, [26] use the continuous PPO algorithm for deriving a control policy for a micro grid management.…”

“…As described in Ji et al [26], the discretization of actions can degrade the performance of the EM and becomes unfeasible with higher dimensionality of the action space. Therefore, [26] use the continuous PPO algorithm for deriving a control policy for a micro grid management. However, in contrast to the investigations described in this paper, the temporal resolution of the simulation in [26] is much broader with one hour time intervals.…”

Optimal Dispatching Strategy of Active Distribution Network for Promoting Local Consumption of Renewable Energy

“…Bi et al (2020) proposed a learning-based dispatching strategy of MGs with RES and ESS. Ji et al (2021) proposed a continuous-control, deep reinforcement learning-based online scheduling method for MGs. Dubuisson et al (2020) proposed a bacterial foraging optimization algorithm for power management in stand-alone MGs.…”