Multi-agent reinforcement learning for modeling and control of thermostatically controlled loads

Abstract

Increasing energy efficiency of thermostatically controlled loads has the potential to substantially reduce domestic energy demand. However, optimizing the efficiency of thermostatically controlled loads requires either an existing model or detailed data from sensors to learn it online. Often, neither is practical because of real-world constraints. In this paper, we demonstrate that this problem can benefit greatly from multi-agent learning and collaboration. Starting with no thermostatically controlled load specific information, the multi-agent modelling and control framework is evaluated over an entire year of operation in a large scale pilot in The Netherlands, constituting over 50 houses, resulting in energy savings of almost 200 kWh per household (or 20% of the energy required for hot water production). Theoretically, these savings can be even higher, a result also validated using simulations. In these experiments, model accuracy in the multi-agent frameworks scales linearly with the number of agents and provides compelling evidence for increased agency as an alternative to additional sensing, domain knowledge or data gathering time. In fact, multi-agent systems can accelerate learning of a thermostatically controlled load's behaviour by multiple orders of magnitude over single-agent systems, enabling active control faster. These findings hold even when learning is carried out in a distributed manner to address privacy issues arising from multi-agent cooperation.

Reinforcement learning for building controls: The opportunities and challenges

“…Chen et al (2018) [66] benchmarked their controller with a "rule-based heuristic" control strategy. Kazmi et al (2019) [67] used a rule-based dead-band controller as the benchmark. Ahn and Park (2019) [68] claimed their controller saved 15.7% energy compared with the fixed pre-determined schedule on OA damper position and temperature setpoint.…”

Research on hierarchical control and optimisation learning method of multi‐energy microgrid considering multi‐agent game

“…Boltzmann probability distribution method [22] is used in this paper to describe the transition probability of the state in the evolutionary game. The Boltzmann probability distribution method selects the action by probability, and the probability of selecting the action a i in the state s is p(a i ) = e Q(s, a i )/λ ∑ a ∈ A e Q(s, a)/λ (16) where λ is the exponential function of the iteration period-k in evolutionary game. When λ increases, the agent's decision randomness also increases; and when λ decreases, the decision randomness also decreases.…”

Multi-Agent Reinforcement Learning Approach for Residential Microgrid Energy Scheduling

“…Moreover, in this paper, a MARL method is used for sequential decision making in multi-agent environment where traditional SARL is difficult to deal with. MARL has been adopted in some fields, such as vehicle routing problem [24] and thermostatically loads modeling [25]. The most universal MARL is equilibrium-based MARL, whose framework accords with Markov games and the evaluation of the learning process is based on all agents' joint behaviors, the equilibrium concept from game theory is introduced to denote optimal joint action [26][27][28][29][30].…”