Multi-Agent Deep Reinforcement Learning for HVAC Control in Commercial Buildings

Deep Reinforcement Learning for Smart Home Energy Management

“…Since DRL problems are mainly based on Markov Decision Process (MDP) framework or its variants (e.g., Partially observable MDP [30], Markov games [17]), we first introduce the background of MDP. Typically, an MDP is defined by a fivetuple (S, A, P, R, γ), where S and A denote the sets of state and action, respectively.…”

“…Since there are many DRL methods in the [47]. For most of existing works on DRL for building energy systems, model-free methods have been used and can be further classified into several types as in [48] [63], Advantage Actor-Critic (A2C) [64], Asynchronous Advantage Actor-Critic (A3C) [65]), and maximum entropy methods (e.g., Multi-Actor Attention-Critic (MAAC) [17], Entropy-Based Collective Advantage Actor-Critic (EB-C-A2C) [27], Entropy-Based Collective Deep Q-Network (EB-C-DQN) [27]). In above-mentioned methods, Q-learning methods do not support continuous actions.…”

“…In model-based methods, DRL agents need to learn building environment models based on historical data Long Short-Term Memory-Deep Deterministic Policy Gradients (LSTM-DDPG) [46], differentiable mode policy-Proximal Policy Optimization (differentiable MPC policy-PPO) [47]. [63], Advantage [64], Asynchronous Advantage Actor-Critic (A3C) [65]), and maximum entropy methods (e.g., Multi-Ac (MAAC) [17], Entropy-Based Collective Advantage Actor-Critic (EB-C-A2C) [27], Entropy-Based Collecti (EB-C-DQN) [27]). In above-mentioned methods, Q-learning methods do not support continuous actions.…”

“…Secondly, there are many sources of uncertainties related to SBEM [16], e.g., renewable generation output, electricity price, indoor temperature, outdoor temperature, CO 2 concentration, number of occupants and power demand of appliances. Thirdly, there are many temporally and spatially coupled operational constraints related to energy subsystems [17] [18], e.g., Heating, Ventilation, and Air Conditioning (HVAC) systems, Energy Storage Systems (ESSs), Electric Vehicles (EVs), which means that the current system decision will affect the future decisions and the decisions among different subsystems should be coordinated. Fourthly, it is difficult to solve large-scale building energy optimization problems in real-time when traditional optimization methods are adopted [19].…”

“…• DRL methods can operate in an online way without knowing any forecast information or statistics information of building environments, which can effectively over- come the challenges brought by system uncertainties and temporally-coupled constraints related to HVAC systems, ESSs, EVs and so on [16] [25]. • Multi-agent DRL methods support the flexible coordination among different building energy subsystems, which can deal with spatially-coupled operational constraints very well [17]. • DRL methods can support "end-to-end" control for largescale building energy optimization problems.…”