Optimal Tap Setting of Voltage Regulation Transformers Using Batch Reinforcement Learning

Abstract

In this paper, we address the problem of setting the tap positions of load tap changers (LTCs) for voltage regulation in radial power distribution systems under uncertain load dynamics. The objective is to find a policy to determine the tap positions that only uses measurements of voltage magnitudes and topology information so as to minimize the voltage deviation across the system. We formulate this problem as a Markov decision process (MDP), and propose a batch reinforcement learning (RL) algorithm to solve it. By taking advantage of a linearized power flow model, we propose an effective algorithm to estimate the voltage magnitudes under different tap settings, which allows the RL algorithm to explore the state and action spaces freely offline without impacting the system operation. To circumvent the "curse of dimensionality" resulted from the large state and action spaces, we propose a sequential learning algorithm to learn an action-value function for each LTC, based on which the optimal tap positions can be directly determined. The effectiveness of the proposed algorithm is validated via numerical simulations on the IEEE 13-bus and 123-bus distribution test feeders.

Coordination of PV Smart Inverters Using Deep Reinforcement Learning for Grid Voltage Regulation

“…A distributed Q-learning is implemented to coordinate generators, OLTCs, and capacitors in [14] for optimal reactive power dispatch. Batch reinforcement learning is applied to achieve cooperation of OLTCs for voltage regulation [15]. Coordination between OLTCs and capacitors are studied with policy gradient method for voltage violation mitigation and operation cost reduction [16].…”

Two-Timescale Voltage Control in Distribution Grids Using Deep Reinforcement Learning

“…The procedure of leveraging this relaxed set (instead of the nonconvex one) is known as SOCP relaxation [32]. Interestingly, it has been shown that under certain conditions, SOCP relaxation is exact in the sense that the set of inequalties (6) holds with equalities at the optimum [33]. Given the capacitor configurationŷ y y(τ ) found at the end of the last interval τ − 1, under the aforementioned relaxed grid model, the voltage regulation on the fast timescale based on the exact AC model can be described as follows…”

“…However, these approaches can be computationally demanding, and do not guarantee optimal performance. A batch reinforcement learning (RL) scheme based on linear function approximation was lately advocated in [6].…”

“…Wedding deep learning with RL (using a DNN to approximate the action-value function), deep (D) RL has offered artificial agents with human-level performance across diverse application domains [18], [20]. (D)RL algorithms have also shown great potential in several challenging power systems control and monitoring tasks [21], [22], [6], [23], [24], [25], and load control [26], [27]. A batch RL scheme using linear function approximation was developed for voltage regulation in distribution systems [6].…”

“…(D)RL algorithms have also shown great potential in several challenging power systems control and monitoring tasks [21], [22], [6], [23], [24], [25], and load control [26], [27]. A batch RL scheme using linear function approximation was developed for voltage regulation in distribution systems [6]. For voltage control of transmission networks, DRL was recently investigated to adjust generator voltage setpoints [21].…”