Safe Off-Policy Deep Reinforcement Learning Algorithm for Volt-VAR Control in Power Distribution Systems

Wang, Wei; Yu, Nanpeng; Gao, Yuanqi; Shi, Jie

doi:10.1109/tsg.2019.2962625

Cited by 206 publications

(115 citation statements)

References 34 publications

(43 reference statements)

Supporting

Mentioning

112

Contrasting

Unclassified

Order By: Relevance

“…This indicates that the fitting of the value function in the proposed method was more robust to the uncertainty of network training. This phenomenon is similar to that reported by Wang et al [33], where past experience was used to improve the actor-critic algorithm's parameter update direction. The result of removing off-policy re-weighting revealed that data from past interactions with the environment are also favorable for AMPI-based reinforcement learning.…”

Section: Data Efficiency Verification During Adaptability To Changes In Vehicle Modelsupporting

confidence: 87%

Data Efficient Reinforcement Learning for Integrated Lateral Planning and Control in Automated Parking System

Song

Chen

Sun

et al. 2020

Sensors

View full text Add to dashboard Cite

Reinforcement learning (RL) is a promising direction in automated parking systems (APSs), as integrating planning and tracking control using RL can potentially maximize the overall performance. However, commonly used model-free RL requires many interactions to achieve acceptable performance, and model-based RL in APS cannot continuously learn. In this paper, a data-efficient RL method is constructed to learn from data by use of a model-based method. The proposed method uses a truncated Monte Carlo tree search to evaluate parking states and select moves. Two artificial neural networks are trained to provide the search probability of each tree branch and the final reward for each state using self-trained data. The data efficiency is enhanced by weighting exploration with parking trajectory returns, an adaptive exploration scheme, and experience augmentation with imaginary rollouts. Without human demonstrations, a novel training pipeline is also used to train the initial action guidance network and the state value network. Compared with path planning and path-following methods, the proposed integrated method can flexibly co-ordinate the longitudinal and lateral motion to park a smaller parking space in one maneuver. Its adaptability to changes in the vehicle model is verified by joint Carsim and MATLAB simulation, demonstrating that the algorithm converges within a few iterations. Finally, experiments using a real vehicle platform are used to further verify the effectiveness of the proposed method. Compared with obtaining rewards using simulation, the proposed method achieves a better final parking attitude and success rate.

show abstract

Section: Data Efficiency Verification During Adaptability To Changes In Vehicle Modelsupporting

confidence: 87%

Data Efficient Reinforcement Learning for Integrated Lateral Planning and Control in Automated Parking System

Song

Chen

Sun

et al. 2020

Sensors

View full text Add to dashboard Cite

show abstract

“…Therefore, the learned strategy may not be feasible in practice. To solve this problem, [46] proposes a volt-var control strategy of distribution network based on safe off-policy DRL algorithm. The volt-var control problem is first modeled as a constrained MDP.…”

Section: A Optimization Of Smart Power and Energy Distribution Grid mentioning

confidence: 99%

Reinforcement Learning and Its Applications in Modern Power and Energy Systems: A Review

Cao

Zhao

et al. 2020

Journal of Modern Power Systems and Clean Energy

233

View full text Add to dashboard Cite

With the growing integration of distributed energy resources (DERs), flexible loads, and other emerging technologies, there are increasing complexities and uncertainties for modern power and energy systems. This brings great challenges to the operation and control. Besides, with the deployment of advanced sensor and smart meters, a large number of data are generated, which brings opportunities for novel data-driven methods to deal with complicated operation and control issues. Among them, reinforcement learning (RL) is one of the most widely promoted methods for control and optimization problems. This paper provides a comprehensive literature review of RL in terms of basic ideas, various types of algorithms, and their applications in power and energy systems. The challenges and further works are also discussed.

show abstract

“…The basic idea is to introduce some penalty terms corresponding to the security constraints, and minimize them in priority during the learning process. Reference [61] adopted this idea to consider charging constraints of electric vehicle batteries.Reference [62] optimized voltage and reactive power by a safe off-policy deep reinforcement learning algorithm to avoid voltage violations.…”

Section: Category 3 Surrogate Modelmentioning

confidence: 99%

Review of Learning-Assisted Power System Optimization

Ruan

Zhong

Zhang

et al. 2020

Preprint

View full text Add to dashboard Cite

Machine learning, with a dramatic breakthrough in recent years, is showing great potential to upgrade the power system optimization toolbox. Understanding the strength and limitation of machine learning approaches is crucial to answer when and how to integrate them in various power system optimization tasks. This paper pays special attention to the coordination between machine learning approaches and optimization models, and carefully evaluates to what extent such data-driven analysis may benefit the rule-based optimization. A series of typical references are selected and categorized into four kinds: the boundary parameter improvement, the optimization option selection, the surrogate model and the hybrid model. This taxonomy provides a novel perspective to understand the latest research progress and achievements. We further discuss several key challenges and provide an in-depth comparison on the features and designs of different categories. Deep integration of machine learning approaches and optimization models is expected to become the most promising technical trend.

show abstract

Safe Off-Policy Deep Reinforcement Learning Algorithm for Volt-VAR Control in Power Distribution Systems

Cited by 206 publications

References 34 publications

Data Efficient Reinforcement Learning for Integrated Lateral Planning and Control in Automated Parking System

Data Efficient Reinforcement Learning for Integrated Lateral Planning and Control in Automated Parking System

Reinforcement Learning and Its Applications in Modern Power and Energy Systems: A Review

Review of Learning-Assisted Power System Optimization

Contact Info

Product

Resources

About