Q-Learning-Based Operation Strategy for Community Battery Energy Storage System (CBESS) in Microgrid System

Abstract

Energy management systems (EMSs) of microgrids (MGs) can be broadly categorized as centralized or decentralized EMSs. The centralized approach may not be suitable for a system having several entities that have their own operation objectives. On the other hand, the use of the decentralized approach leads to an increase in the operation cost due to local optimization. In this paper, both centralized and decentralized approaches are combined for managing the operation of a distributed system, which is comprised of an MG and a community battery storage system (CBESS). The MG is formed by grouping all entities having the same operation objective and is operated under a centralized controller, i.e., a microgrid EMS (MG-EMS). The CBESS is operated by using its local controller with different operation objectives. A Q-learning-based operation strategy is proposed for optimal operation of CBESS in both grid-connected and islanded modes. The objective of CBESS in the grid-connected mode is to maximize its profit while the objective of CBESS in islanded mode is to minimize the load shedding amount in the entire system by cooperating with the MG. A comparison between the Q-learning-based strategy and a conventional centralized-based strategy is presented to show the effectiveness of the proposed strategy. In addition, an adjusted epsilon is also introduced for epsilon-greedy policy to reduce the learning time and improve the operation results.

Reinforcement-Learning-Based Virtual Energy Storage System Operation Strategy for Wind Power Forecast Uncertainty Management

“…In the RL approach, the state-action value function is estimated by learning. In widely used Q-learning-based RL approaches [26][27][28][29], the state-action value function is estimated as…”

Profit Maximizing Control of a Microgrid with Renewable Generation and BESS Based on a Battery Cycle Life Model and Energy Price Forecasting

“…Given proper control of storage units and communications with the electricity market, the non-dispatchable renewable power can be smoothed and used on demand, therefore reducing the difficulty of power scheduling in the main grid operation [3]. In many studies that focus on the control of renewable power systems [4][5][6][7][8][9][10][11][12][13][14][15], the battery energy storage system (BESS) is essential for controlling the actual power dispatched to the local customers and the grid. Utilizing forecasting data on renewable power and power demand to arrange BESS actions over different periods, the power constraints in the microgrid and the operating parameters in the main grid can be satisfied.…”

“…Many control strategies and optimization methods, including model predictive control (MPC) [5,8,14,15], dynamic programming (DP) [3,4,7,16], sliding mode control (SMC) [17,18], reinforcement learning (RL) [9,10], particle swarm optimization (PSO) [11,[19][20][21], and mixed-integer linear programming (MILP) [6,13,15], have been proposed for renewable power control under different conditions. The use of MPC is mainly due to forecasting errors.…”

“…Authors of [9] formulate the energy management problem in a microgrid as a Markov decision process to minimize the daily cost and address it with deep learning. A Q-learning based operation strategy to maximize the profit of a microgrid with community battery storage system is proposed in [10], which combines both centralized and decentralized approaches to control the system units.…”

“…The number of charge-discharge cycles that the battery can experience will be counted, and the effects of stress factors, such as temperature, depth of discharge (DOD), average state of charge (SOC), and operating currents on the rate of life cycle loss can be investigated. It should be pointed out that most of the cost models of BESS in previously reviewed studies are either not considered [3,[5][6][7][8][9][10][14][15][16] in the optimization problem, or simple models that without consideration on the cycling model under different stress factors [4,11,13].…”