Federated Multiagent Deep Reinforcement Learning Approach via Physics-Informed Reward for Multimicrogrid Energy Management

Abstract

The utilization of large-scale distributed renewable energy promotes the development of the multi-microgrid (MMG), which raises the need of developing an effective energy management method to minimize economic costs and keep self energy-sufficiency. The multi-agent deep reinforcement learning (MADRL) has been widely used for the energy management problem because of its real-time scheduling ability. However, its training requires massive energy operation data of microgrids (MGs), while gathering these data from different MGs would threaten their privacy and data security. Therefore, this paper tackles this practical yet challenging issue by proposing a federated multi-agent deep reinforcement learning (F-MADRL) algorithm via the physics-informed reward. In this algorithm, the federated learning (FL) mechanism is introduced to train the F-MADRL algorithm thus ensures the privacy and the security of data. In addition, a decentralized MMG model is built, and the energy of each participated MG is managed by an agent, which aims to minimize economic costs and keep self energysufficiency according to the physics-informed reward. At first, MGs individually execute the self-training based on local energy operation data to train their local agent models. Then, these local models are periodically uploaded to a server and their parameters are aggregated to build a global agent, which will be broadcasted to MGs and replace their local agents. In this way, the experience of each MG agent can be shared and the energy operation data is not explicitly transmitted, thus protecting the privacy and ensuring data security. Finally, experiments are conducted on Oak Ridge national laboratory distributed energy control communication lab microgrid (ORNL-MG) test system, and the comparisons are carried out to verify the effectiveness of introducing the FL mechanism and the outperformance of our proposed F-MADRL.

A Survey on Physics Informed Reinforcement Learning: Review and Open Problems

“…Physics values extracted/ derived from the environ-ment or system has been directly used by RL agents in form of physics parameters [113], dynamic movement (physics) primitives [3], physical state [59] and physical target [75]. For example in [75], the reward is created to meet two physical objectives/ targets: operation cost and self-energy sustainability. In an adaptive cruise control problem [59], authors use desired physical parameters e.g.…”

“…A well-defined reward function is crucial for successful reinforcement learning, PIRL approaches also seek to incorporate physical constraints into the design for safe learning and more efficient reward functions. For example, in [68] the designed reward incorporates IMU sensor data, imbibing inertial constraints, while in [75] the physics informed reward is designed to satisfy explicit operational targets. To ensure safe exploration during training and deployment, works such as [133,141] learn a data-driven barrier certificate based on physical property-based losses and a set of unsafe state vectors.…”

Joint scheduling optimization of a microgrid with integration of renewable energy sources and electric vehicles considering energy and reserve minimization

“…Equations ( 4) and (5) show the lower and upper boundaries of the available energy in the EV's battery, respectively.…”

“…Existing technologies need to be rethought covering environmental concerns, 4 since the power system, including smart grid, accounts for the lion's share of global emissions. 5 As a consequence, the electricity sector requires improved and more efficient scheduling technologies for allocating available resources. 6 PEVs and plug-in hybrid electric vehicles (PHEVs) are another topic receiving more attention because of the lessened environmental impact that they have.…”

“…In this way, one of the most important aspects of MG optimization is finding a solution to the environmental/economic dispatch dilemma. Existing technologies need to be rethought covering environmental concerns, 4 since the power system, including smart grid, accounts for the lion's share of global emissions 5 . As a consequence, the electricity sector requires improved and more efficient scheduling technologies for allocating available resources 6 …”