Adaptive Power System Emergency Control Using Deep Reinforcement Learning

Abstract

Power system emergency control is generally regarded as the last safety net for grid security and resiliency. Existing emergency control schemes are usually designed off-line based on either the conceived "worst" case scenario or a few typical operation scenarios. These schemes are facing significant adaptiveness and robustness issues as increasing uncertainties and variations occur in modern electrical grids. To address these challenges, this paper developed novel adaptive emergency control schemes using deep reinforcement learning (DRL) by leveraging the high-dimensional feature extraction and non-linear generalization capabilities of DRL for complex power systems.Furthermore, an open-source platform named Reinforcement Learning for Grid Control (RLGC) has been designed for the first time to assist the development and benchmarking of DRL algorithms for power system control. Details of the platform and DRL-based emergency control schemes for generator dynamic braking and under-voltage load shedding are presented. Robustness of the developed DRL method to different simulation scenarios, model parameter uncertainty and noise in the observations is investigated. Extensive case studies performed in both the two-area, four-machine system and the IEEE 39-bus system have demonstrated excellent performance and robustness of the proposed schemes.

Proactive Resilience of Power Systems Against Natural Disasters: A Literature Review

“…Unlike the optimization-based methods that obtain the intermediate results and final solutions as a whole, which are either based on day/hour-ahead framework or real-time framework (including two-stage framework), machine learning-based methods could sperate the training process of intermediate results and problem-solving process in different time horizons. For example, in the work [127], a deep reinforcement learning (DRL) and simulation-based framework with a pre-calculated neural network structure are used to deal with an adaptive power system emergency control problem. The reinforcement learning module, as a problem-solving engine, automatically determines its Q-network weight parameters via repeatedly simulated samples before the actual decision-making of a proactive strategy.…”

(Deep) Reinforcement learning for electric power system control and related problems: A short review and perspectives

“…The aim of controls is to keep the system in synchronism (with angular velocities equal or very close to the nimnal value defined by the nominal system frequency). References [17,[58][59][60][61][62][63][64]73] dealt with transient instability control by controlling individual electric power system components such as thyristor-controlled series capacitor [17,58,59] and a dynamic brake (a resistor usually located near electricity generation plant) to absorb excess of electricity generation [60,61]. Q-learning was used in [17,[60][61][62] while [58,59] suggested fitted Q-iteration.…”

“…Inclusion of state history to recover Markov property in partially observable problems was considered in [62]. A dynamic brake wa also considered in [73] for emergency control using DRL (DQN was an approach of the choice in [73]) largely following implementation details presented in [17,62]. The problem of transient angle instability was also considered within wide-area control systems [63,64].…”

“…Reference [73]) dealt with Fault Induced Delayed Voltage Recovery (FIDVR) problem through DRL (DQN) where under-voltage load shedding was used as an emergency control. FIDVR problem is caused by a fault in transmission system resulting in slow voltage recovery in distribution system (the problem is connected to presence of reactive power loads in distribution and increased demand for this power due to reduced voltage causing slow recovery).…”

Probability-Based Energy Reinforced Management of Electric Vehicle Aggregation in the Electrical Grid Frequency Regulation

“…Unlike traditional reinforcement learning, the DRL algorithms use powerful deep neuron networks to approximate their value function (such as Q-table), enabling automatic high-dimensional feature extraction and end-toend learning. Recently, the advantages of DRL were recognized by the community and some attempts were made to leverage DRL in various applications for electrical grid, including operational control [21]- [24], electricity market [25], [26], demand response [27] and energy management [28]. Although these applications presented advantageous results in their respective fields, several challenges were encountered.…”