Comparing Policy Gradient and Value Function Based Reinforcement Learning Methods in Simulated Electrical Power Trade

Abstract

This version is available at https://strathprints.strath.ac.uk/33071/ Strathprints is designed to allow users to access the research output of the University of Strathclyde. Unless otherwise explicitly stated on the manuscript, Copyright © and Moral Rights for the papers on this site are retained by the individual authors and/or other copyright owners. Please check the manuscript for details of any other licences that may have been applied. You may not engage in further distribution of the material for any profitmaking activities or any commercial gain. You may freely distribute both the url (https://strathprints.strath.ac.uk/) and the content of this paper for research or private study, educational, or not-for-profit purposes without prior permission or charge.Any correspondence concerning this service should be sent to the Strathprints administrator: strathprints@strath.ac.ukThe Strathprints institutional repository (https://strathprints.strath.ac.uk) is a digital archive of University of Strathclyde research outputs. It has been developed to disseminate open access research outputs, expose data about those outputs, and enable the management and persistent access to Strathclyde's intellectual output. Abstract-In electrical power engineering, reinforcement learning algorithms can be used to model the strategies of electricity market participants. However, traditional value function based reinforcement learning algorithms suffer from convergence issues when used with value function approximators. Function approximation is required in this domain to capture the characteristics of the complex and continuous multivariate problem space. The contribution of this paper is the comparison of policy gradient reinforcement learning methods, using artificial neural networks for policy function approximation, with traditional value function based methods in simulations of electricity trade. The methods are compared using an AC optimal power flow based power exchange auction market model and a reference electric power system model.

A deep asynchronous actor‐critic learning‐based event‐triggered decentralized load frequency control of power systems with communication delays

“…In addition, small changes in the value function will cause huge changes to the policy, thereby affecting convergence. In policy‐based methods, see, for example, the policy gradient method, 41 it directly learns the policy by parameterizing the policy. It is suitable for high‐dimensional continuous action space and stochastic policy, but the disadvantage is that the variance of the gradient estimation is relatively high, and it is easy to converge to the non‐optimal policy.…”

Reinforcement Learning Techniques for Optimal Power Control in Grid-Connected Microgrids: A Comprehensive Review

“…Since ( , ) is also unknown, it may also be estimated or better represented by an N-step return, i.e., the total expected discounted reward for N stages (say taking 5 future states). This is possible because the absolute value of ( , ) is not needed but how much better it is than the current policy [79]. Using (14), the ANN can be trained to adjust its parameters in the direction of better policy performance using the gradient ascent method.…”

Performance Improvement of Linux CPU Scheduler Using Policy Gradient Reinforcement Learning for Android Smartphones

“…The policy gradient reinforcement learning implemented by artificial neural network for the simulated electric power trade shows the result of maximizing the performance [23]. In addition, a study that reduced CPU power consumption in mobile environment by learning through artificial neural network was introduced [24] and showed the possibility of OS optimization through machine learning.…”