Accelerated Derivative-Free Deep Reinforcement Learning for Large-Scale Grid Emergency Voltage Control

Huang, Renke; Chen, Yujiao; Yin, Tianzhixi; Li, Xinya; Li, Ang; Tan, Jie; Yu, Wenhao; Liu, Yuan; Huang, Qiuhua

doi:10.1109/tpwrs.2021.3095179

Cited by 40 publications

(30 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our previous work, we have designed a deep Q learning based RL control [6] for emergency load shedding to address this problem. Subsequently, we developed an accelerated Augmented Random Search (ARS) algorithm in [8] to quickly train the neural network-based control policy in a parallel computing mechanism. In the ARS algorithm in [8], the ARS agent performs parameter-space exploration and estimates the gradient of the expected reward using sampled rollouts to update the control policy.…”

Section: Reinforcement Learning-based Emergency Load Sheddingmentioning

confidence: 99%

“…We demonstrate the effectiveness of the proposed safe ARSbased emergency load shedding on the 39-bus IEEE testcase in Section IV. Furthermore, we show that in comparison to the standard ARS-based load shedding [8], the Barrier function-based safe ARS load shedding can be more adaptive to contingencies not seen in the training, indicating its promise in ensuring the safe operation of actual power systems in deployment.…”

Section: Introductionmentioning

confidence: 95%

“…Recently, reinforcement learning (RL) based approach has been successfully developed to solve the emergency voltage control problem [6]- [8]. In the RL approach, the machine agent interacts with the system/environment, observes the resulting system state and the corresponding reward (which is suitably defined to encourage the voltage stability constraint), and updates the control actions in a way to maximize the reward [6].…”

Section: Introductionmentioning

confidence: 99%

“…In the RL approach, the machine agent interacts with the system/environment, observes the resulting system state and the corresponding reward (which is suitably defined to encourage the voltage stability constraint), and updates the control actions in a way to maximize the reward [6]. In comparison to the rule-based UVLS, the RLbased load shedding can significantly reduce the amount of loads that need to be shed in order to recover the voltage stability of the system [8]. Dominantly considered in the existing literature, Markov decision process (MDP) based framework was utilized to develop a varieties of RL algorithms, many of those were investigated for solving emergency voltage control problem.…”

Section: Introductionmentioning

confidence: 99%

“…In [6], we have designed a deep Q learning based RL control for emergency load shedding in response to voltage stability issues of the electric power grid. Subsequently, an accelerated Augmented Random Search (ARS) algorithm was introduced in [8] to quickly train the neural network-based control policy in a parallel computing mechanism. This ARS method, sketched in Section II, is implemented on a high-performance computing platform in a novel nested parallel architecture using the Ray framework.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Barrier Function-based Safe Reinforcement Learning for Emergency Control of Power Systems

Mukherjee

Huang

et al. 2021

2021 60th IEEE Conference on Decision and Control (CDC)

Self Cite

View full text Add to dashboard Cite

Under voltage load shedding has been considered as a standard and effective measure to recover the voltage stability of the electric power grid under emergency and severe conditions. However, this scheme usually trips a massive amount of load which can be unnecessary and harmful to customers. Recently, deep reinforcement learning (RL) has been regarded and adopted as a promising approach that can significantly reduce the amount of load shedding. However, like most existing machine learning (ML)-based control techniques, RL control usually cannot guarantee the safety of the systems under control. In this paper, we introduce a novel safe RL method for emergency load shedding of power systems, that can enhance the safe voltage recovery of the electric power grid after experiencing faults. Unlike the standard RL method, the safe RL method has a reward function consisting of a Barrier function that goes to minus infinity when the system state goes to the safety bounds. Consequently, the optimal control policy can render the power system to avoid the safety bounds. This method is general and can be applied to other safety-critical control problems. Numerical simulations on the 39-bus IEEE benchmark is performed to demonstrate the effectiveness of the proposed safe RL emergency control, as well as its adaptive capability to faults not seen in the training.

show abstract

Section: Reinforcement Learning-based Emergency Load Sheddingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 95%