Smart Grid for Industry Using Multi-Agent Reinforcement Learning

Abstract

The growing share of renewable power generation leads to increasingly fluctuating and generally rising electricity prices. This is a challenge for industrial companies. However, electricity expenses can be reduced by adapting the energy demand of production processes to the volatile prices on the markets. This approach depicts the new paradigm of energy flexibility to reduce electricity costs. At the same time, using electricity self-generation further offers possibilities for decreasing energy costs. In addition, energy flexibility can be gradually increased by on-site power storage, e.g., stationary batteries. As a consequence, both the electricity demand of the manufacturing system and the supply side, including battery storage, self-generation, and the energy market, need to be controlled in a holistic manner, thus resulting in a smart grid solution for industrial sites. This coordination represents a complex optimization problem, which additionally is highly stochastic due to unforeseen events like machine breakdowns, changing prices, or changing energy availability. This paper presents an approach to controlling a complex system of production resources, battery storage, electricity self-supply, and short-term market trading using multi-agent reinforcement learning (MARL). The results of a case study demonstrate that the developed system can outperform the rule-based reactive control strategy (RCS) frequently used. Although the metaheuristic benchmark based on simulated annealing performs better, MARL enables faster reactions because of the significantly lower computation costs for its own execution.

Exploiting Battery Storages With Reinforcement Learning: A Review for Energy Professionals

“…Muriithi & Chowdhury [146] capture the degradation of a lithium-ion battery in terms of depth of discharge. Roesch et al [125] use a sophisticated battery degradation model to capture the impacts of irregular charging and discharging cycles on battery degradation. Yang et al [17] penalize the number of switches between charging, idle and discharging modes.…”

“…Batteries are emerging as an element of factory energy systems, either for rescheduling production tasks to lower electricity price periods [125] or to ensure the continuity of production during outages [113].…”

Distributed Reinforcement Learning for the Management of a Smart Grid Interconnecting Independent Prosumers

“…This synthetic review on related works leads to one observation: very few works focused on a general smart grid model, in which consumers may or may not have power production capabilities and power storage facilities, and considered learning approaches. To our knowledge, only [27] proposed a multi-agent reinforcement-learning approach to controlling a smart grid composed of production resources, battery storage, electricity self-supply, and short-term market trading. Event though the proposed algorithm offers a significantly high computation speed, it has local optima issues that cause it to be outperformed by the use of a simulated annealing in terms of energy costs.…”

Disentangling Successor Features for Coordination in Multi-agent Reinforcement Learning

“…Multi-agent reinforcement learning (MARL) has shown promising results for applications including traffic control [20], smart grids [31,14], autonomous driving [27], and UAV control [44]. Many existing approaches implement centralized architectures to ensure coordination among agents [20,36,18].…”