Load Frequency Control: A Deep Multi-Agent Reinforcement Learning Approach

Rozada, Sergio; Apostolopoulou, Dimitra; Alonso, Eduardo

doi:10.1109/pesgm41954.2020.9281614

Cited by 21 publications

(19 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Implicit cooperation, which keeps personal information at the local level while encouraging cooperation towards global objectives, has been thus far under-researched beyond frequency control. In [49], agents learn the optimal way of acting and interacting with the environment to restore frequency using local information only. This is a promising approach for decentralised control.…”

Section: Marl-based Energy Coordination: Literature Review and Gap An...mentioning

confidence: 99%

Scalable multi-agent reinforcement learning for distributed control of residential energy flexibility

Charbonnier,

Morstyn,

McCulloch

2022

Preprint

View full text Add to dashboard Cite

This paper proposes a novel scalable type of multi-agent reinforcement learning-based coordination for distributed residential energy. Cooperating agents learn to control the flexibility offered by electric vehicles, space heating and flexible loads in a partially observable stochastic environment. In the standard independent Q-learning approach, the coordination performance of agents under partial observability drops at scale in stochastic environments.Here, the novel combination of learning from off-line convex optimisations on historical data and isolating marginal contributions to total rewards in reward signals increases stability and performance at scale. Using fixed-size Q-tables, prosumers are able to assess their marginal impact on total system objectives without sharing personal data either with each other or with a central coordinator. Case studies are used to assess the fitness of different combinations of exploration sources, reward definitions, and multi-agent learning frameworks. It is demonstrated that the proposed strategies create value at individual and system levels thanks to reductions in the costs of energy imports, losses, distribution network congestion, battery depreciation and greenhouse gas emissions.

show abstract

Section: Marl-based Energy Coordination: Literature Review and Gap An...mentioning

confidence: 99%

Scalable multi-agent reinforcement learning for distributed control of residential energy flexibility

Charbonnier,

Morstyn,

McCulloch

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…As such, one agent learns to balance the entire system while the others have zero output. Further analysis on secondary control using Model I may be found in [21].…”

Section: A Secondary Control -Model Imentioning

confidence: 99%

“…The main drawback of these methods is their computational complexity, that grows exponentially in the number of agents. However, the rise of Deep Learning has opened the door to new techniques and algorithms that address these scalability issues in the load frequency control problem (see, e.g., [21], [22]).…”

Section: Introductionmentioning

confidence: 99%

“…To summarise, the contributions of the paper are: i) reformulation of the load frequency control problem as a Markov Decision Process (MDP); ii) use of a detailed model taking into account the network, renewable-based generation and generator rate constraints; iii) design of the reward function of the agents so that frequency deviation and total cost are minimised; iv) development of the proposed framework to solve the optimal load frequency control in a fully distributed manner with only the use of local information; and v) validation of its robustness against uncertainty introduced from renewable-based generation. This problem was initially introduced in [21] and is extended in this paper to implement tertiary control or economic dispatch, i.e., the generation units modify their output to meet the change in load in a cost efficient way; to include a detailed power systems' model by explicitly incorporating the network, windbased generation and a more realistic model of synchronous generator by including the generator rate constraints (GRC).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Deep Multi-Agent Reinforcement Learning for Cost Efficient Distributed Load Frequency Control

Rozada¹,

Apostolopoulou²,

Alonso³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

The rise of microgrid-based architectures is heavily modifying the energy control landscape in distribution systems making distributed control mechanisms necessary to ensure reliable power system operations. In this paper, we propose the use of Reinforcement Learning techniques to implement load frequency control without requiring a central authority. To this end, we approximate the optimal solution of the primary, secondary, and tertiary control with the use of the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm. Generation units are characterised as agents that learn how to maximise their long-term performance by acting and interacting with the environment to balance generation and load in a cost efficient way. Network effects are also modelled in our framework for the restoration of frequency to the nominal value. We validate our Reinforcement Learning methodology through numerical results and show that it can be used to implement the load frequency control in a distributed and cost efficient way.

show abstract

“…However, the rise of Deep Learning has opened the door to new techniques and algorithms that address these scalability issues in the LFC problem (see e.g. [20,21]).…”

Section: Introductionmentioning

confidence: 99%

Deep multi‐agent Reinforcement Learning for cost‐efficient distributed load frequency control

Rozada

Apostolopoulou

Alonso

2021

IET Energy Syst Integration

Self Cite

View full text Add to dashboard Cite

The rise of microgrid-based architectures is modifying significantly the energy control landscape in distribution systems, making distributed control mechanisms necessary to ensure reliable power system operations. In this article, the use of Reinforcement Learning techniques is proposed to implement load frequency control (LFC) without requiring a central authority. To this end, a detailed model of power system dynamic behaviour is formulated by representing individual generator dynamics, generator rate and network constraints, renewable-based generation, and realistic load realisations. The LFC problem is recast as a Markov Decision Process, and the Multi-Agent Deep Deterministic Policy Gradient algorithm is used to approximate the optimal solution of all LFC layers, that is, primary, secondary and tertiary. The proposed LFC framework operates through centralised learning and distributed implementation. In particular, there is no information interchange between generating units during operation. Thus, no communication infrastructure is necessary and information privacy between them is respected. The proposed framework is validated through numerical results and it is shown that it can be used to implement LFC in a distributed and cost-efficient manner.This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

show abstract

Load Frequency Control: A Deep Multi-Agent Reinforcement Learning Approach

Cited by 21 publications

References 15 publications

Scalable multi-agent reinforcement learning for distributed control of residential energy flexibility

Scalable multi-agent reinforcement learning for distributed control of residential energy flexibility

Deep Multi-Agent Reinforcement Learning for Cost Efficient Distributed Load Frequency Control

Deep multi‐agent Reinforcement Learning for cost‐efficient distributed load frequency control

Contact Info

Product

Resources

About