Deep multi‐agent Reinforcement Learning for cost‐efficient distributed load frequency control

Rozada, Sergio; Apostolopoulou, Dimitra; Alonso, Eduardo

doi:10.1049/esi2.12030

Cited by 4 publications

(3 citation statements)

References 42 publications

(69 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A control method for load frequency management has been proposed for all layers of microgrid control with no communication required between operating nodes, providing an extra security layer. This proposed DRL method is implemented with central learning but in a distributed manner [47]. Optimizing a microgrid in real-time is a challenging process that can be achieved using a double-deep Q network-based algorithm [48].…”

Section: Relevant Published Work In the Year 2021mentioning

confidence: 99%

Deep Reinforcement Learning for Control of Microgrids: A Review

Hassan

Farhan

Ahmed

et al. 2022

Research Journal Of Computer Science And Information Technology

View full text Add to dashboard Cite

A microgrid is widely accepted as a prominent solution to enhance resilience and performance in distributed power systems. Microgrids are flexible for adding distributed energy resources in the ecosystem of the electrical networks. Control techniques are used to synchronize distributed energy resources (DERs) due to their turbulent nature. DERs including alternating current, direct current and hybrid load with storage systems have been used in microgrids quite frequently due to which controlling the flow of energy in microgrids have been complex task with traditional control approaches. Distributed as well central approach to apply control algorithms is well-known methods to regulate frequency and voltage in microgrids. Recently techniques based of artificial intelligence are being applied for the problems that arise in operation and control of latest generation microgrids and smart grids. Such techniques are categorized in machine learning and deep learning in broader terms. The objective of this research is to survey the latest strategies of control in microgrids using the deep reinforcement learning approach (DRL). Other techniques of artificial intelligence had already been reviewed extensively but the use of DRL has increased in the past couple of years. To bridge the gap for the researchers, this survey paper is being presented with a focus on only Microgrids control DRL techniques for voltage control and frequency regulation with distributed, cooperative and multi agent approaches are presented in this research.

show abstract

Section: Relevant Published Work In the Year 2021mentioning

confidence: 99%

Deep Reinforcement Learning for Control of Microgrids: A Review

Hassan

Farhan

Ahmed

et al. 2022

Research Journal Of Computer Science And Information Technology

View full text Add to dashboard Cite

show abstract

“…When applied to complex scenarios, such as multimicrogrids with EVs, the 'dimension explosion' problem in the action space may occur. In [31], the LFC problem was reconstructed as a Markov decision process, and a deep deterministic policy gradient (DDPG) algorithm with continuous action space was used to approximate the optimal solution for the LFC layer to realise the frequency control of a single island microgrid. Evidently, this cannot be applied to the coordination of multiple microgrids.…”

Section: Introductionmentioning

confidence: 99%

Load frequency control strategy for islanded multimicrogrids with V2G dependent on learning‐based model predictive control

Fan,

Yang,

et al. 2023

IET Generation Trans & Dist

View full text Add to dashboard Cite

System reliability and stability can be significantly improved by the interconnected operation of multimicrogrids, and electric vehicles (EVs) provide a more flexible solution for frequency control, which also present challenges for frequency control. Therefore, a load frequency control (LFC) strategy for multimicrogrids with vehicle to grid (V2G) dependent on learning‐based model predictive control (MPC) is proposed. First, a controller‐interconnected multimicrogrid topology is proposed; thus, a multimicrogrid consisting of microturbines (MTs), distributed power sources, and EVs and their random power constraints is established. Second, a control parameter adaptive algorithm based on learning‐based MPC is designed. The real‐time frequency offset and EV station output power boundary are used as the state set, adjustable parameters of the MPC controller are used as the action set, and reward function is set with frequency deviation so that the adaptive adjustment of the weight parameters of the MPC controller is realised. Additionally, the improved MPC controller designed in this paper can transform the frequency control process into an optimization problem, which is well adapted to various random constraints in the control process. In addition, the deep deterministic policy gradient (DDPG)‐MPC double‐layer controller can prevent machine learning controller failure. Finally, the simulation results show that, compared with traditional control and MPC algorithms, the learning‐based MPC controller applied to the controller interconnection structure can exchange information between submicrogrids. Moreover, based on the experience accumulated in the prelearning process, the controller parameters can be updated according to the environmental state in real time, thereby significantly improving the robustness and rapidity of the multimicrogrid frequency control process. Meanwhile, compared with a traditional DDPG controller, the proposed controller with double‐layer coupling structure can better ensure the safe operation of the multimicrogrid system when the machine learning agent fails and cannot output actions normally.

show abstract

“…Outstanding research has achieved good evaluations in hybrid environments 7 or multiobjective tasks, 8 but it is only in recent years that MARL has been used in engineering applications. They mainly focus on scheduling and optimization problems in multiple engineering domains, such as traffic control, 9 autonomous driving, 10,11 base station communication, 12 load frequency optimization, 13 electric vehicle charging and discharging planning, 14 power allocation, 15 and so forth. In addition, as more complex problems are considered, more applicable systems are modeled and analyzed, such as Markov Repairable Systems, 16 Markov Jumping Systems, 17 and so forth.…”

Section: Introductionmentioning

confidence: 99%

Multiagent reinforcement learning for strictly constrained tasks based on Reward Recorder

Ding

Yan

Liu

2022

Int J of Intelligent Sys

View full text Add to dashboard Cite

Multiagent reinforcement learning (MARL) has been widely applied in engineering problems. However, many strictly constrained problems such as distributed optimization in engineering applications are still a great challenge to MARL. Especially for strict global constraints of agents' actions, it is very easy to lead to sparse rewards. Besides, existing studies cannot solve the instability caused by partial observability while making the algorithm fully distributed. Algorithms with centralized training may encounter significant obstacles in realworld deployment. For the first time, we provide theoretical analysis for MARL to determine the adverse effects of partial observability on convergence, and a fully distributed and convergent MARL algorithm based on Reward Recorder is proposed. Each agent runs an independent reinforcement learning algorithm and uses the average-consensus protocol to estimate the global state-action value locally to achieve global optimization. To verify the performance of the algorithm, we propose a novel generalized constrained optimization model, which includes local inequality constraints and strict global constraints. The proposed distributed reinforcement learning algorithm is supported by several simulation examples. The results reveal that the proposed algorithm has high stability and excellent decision-making ability.

show abstract

Deep multi‐agent Reinforcement Learning for cost‐efficient distributed load frequency control

Cited by 4 publications

References 42 publications

Deep Reinforcement Learning for Control of Microgrids: A Review

Deep Reinforcement Learning for Control of Microgrids: A Review

Load frequency control strategy for islanded multimicrogrids with V2G dependent on learning‐based model predictive control

Multiagent reinforcement learning for strictly constrained tasks based on Reward Recorder

Contact Info

Product

Resources

About