A review of reinforcement learning methodologies for controlling occupant comfort in buildings

Han, Mengjie; May, Ross; Zhang, Xingxing; Wang, Xinru; Pan, Song; Yan, Da; Yuan, Jiao; Xu, Liguo

doi:10.1016/j.scs.2019.101748

Cited by 119 publications

(51 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They summarized the applications of reinforcement learning in building controls for three specific purposes: occupant comfort, energy savings, and demand response. Han et al (2019) [25] reviewed the application of reinforcement learning for occupant comfort management. Thirty-three empirical studies on this topic have been identified and reviewed.…”

Section: Previous Reviewsmentioning

confidence: 99%

Reinforcement learning for building controls: The opportunities and challenges

2020

View full text Add to dashboard Cite

Building controls are becoming more important and complicated due to the dynamic and stochastic energy demand, on-site intermittent energy supply, as well as energy storage, making it difficult for them to be optimized by conventional control techniques. Reinforcement Learning (RL), as an emerging control technique, has attracted growing research interest and demonstrated its potential to enhance building performance while addressing some limitations of other advanced control techniques, such as model predictive control. This study conducted a comprehensive review of existing studies that applied RL for building controls. It provided a detailed breakdown of the existing RL studies that use a specific variation of each major component of the Reinforcement Learning: algorithm, state, action, reward, and environment. We found RL for building controls is still in the research stage with limited applications (11%) in real buildings. Three significant barriers prevent the adoption of RL controllers in actual building controls: (1) the training process is time consuming and data demanding, (2) the control security and robustness need to be enhanced, and (3) the generalization capabilities of RL controllers need to be improved using approaches such as transfer learning. Future research may focus on developing RL controllers that could be used in real buildings, addressing current RL challenges, such as accelerating training and enhancing control robustness, as well as developing an opensource testbed and dataset for performance benchmarking of RL controllers.

show abstract

Section: Previous Reviewsmentioning

confidence: 99%

Reinforcement learning for building controls: The opportunities and challenges

2020

View full text Add to dashboard Cite

show abstract

“…As shown in the literature, RL has been widely used in solving MDPs [33], [40]- [42]. In a RL process, the agent learns its optimal policy π by interacting with the environment, where a policy π is a mapping from states to probabilities of selecting every possible action [40].…”

Section: B Rlmentioning

confidence: 99%

“…For example, the applications of DRL in power systems, communications and networking, autonomous IoT, cyber security, and multi-agent systems can be found in [20], [29]- [32]. In addition, there are several surveys on building energy systems, but the involved methods are RL [33]- [36] or other artificial intelligence methods (e.g., Model Predictive Control (MPC), Fuzzy Logic (FL)) [37]. Although some DRL algorithms are mentioned in [38] and [39], they mainly focus on different applications (ranging from load forecasting to cyber security) of RL/DRL in sustainable energy and electric systems.…”

Section: Introductionmentioning

confidence: 99%

“…For convenience, we provide the comparison between our work and related surveys in Table I. It can be observed that our work completely focuses on DRL for SBEM from the perspective of system complexities (i.e., a single building energy subsystem, multiple energy subsystems in buildings, and building microgrids), while works in [33]- [37] mainly focus on RL and other artificial intelligence methods for occupant comfort control and building energy optimization. Compared with [39], we provide a deeper analysis of DRL-based building energy optimization.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Deep Reinforcement Learning for Smart Home Energy Management

Xie

et al. 2020

IEEE Internet Things J.

286

View full text Add to dashboard Cite

In this paper, we investigate an energy cost minimization problem for a smart home in the absence of a building thermal dynamics model with the consideration of a comfortable temperature range. Due to the existence of model uncertainty, parameter uncertainty (e.g., renewable generation output, nonshiftable power demand, outdoor temperature, and electricity price) and temporally-coupled operational constraints, it is very challenging to determine the optimal energy management strategy for scheduling Heating, Ventilation, and Air Conditioning (HVAC) systems and energy storage systems in the smart home. To address the challenge, we first formulate the above problem as a Markov decision process, and then propose an energy management strategy based on Deep Deterministic Policy Gradients (DDPG). It is worth mentioning that the proposed strategy does not require the prior knowledge of uncertain parameters and building thermal dynamics model. Simulation results based on real-world traces demonstrate the effectiveness and robustness of the proposed strategy.

show abstract

“…Though the occupants' comfort depends on various factors, it is commonly reduced to thermal comfort, for which HVAC controllers are usually optimized, but air quality is seldom considered [2]. In practice, the carbon dioxide (CO2) levels are commonly used as internal air quality (IAQ) indicators.…”

Section: Introductionmentioning

confidence: 99%

Reinforcement Learning-Based School Energy Management System

2020

View full text Add to dashboard Cite

Energy efficiency is a key to reduced carbon footprint, savings on energy bills, and sustainability for future generations. For instance, in hot climate countries such as Qatar, buildings are high energy consumers due to air conditioning that resulted from high temperatures and humidity. Optimizing the building energy management system will reduce unnecessary energy consumptions, improve indoor environmental conditions, maximize building occupant’s comfort, and limit building greenhouse gas emissions. However, lowering energy consumption cannot be done despite the occupants’ comfort. Solutions must take into account these tradeoffs. Conventional Building Energy Management methods suffer from a high dimensional and complex control environment. In recent years, the Deep Reinforcement Learning algorithm, applying neural networks for function approximation, shows promising results in handling such complex problems. In this work, a Deep Reinforcement Learning agent is proposed for controlling and optimizing a school building’s energy consumption. It is designed to search for optimal policies to minimize energy consumption, maintain thermal comfort, and reduce indoor contaminant levels in a challenging 21-zone environment. First, the agent is trained with the baseline in a supervised learning framework. After cloning the baseline strategy, the agent learns with proximal policy optimization in an actor-critic framework. The performance is evaluated on a school model simulated environment considering thermal comfort, CO2 levels, and energy consumption. The proposed methodology can achieve a 21% reduction in energy consumption, a 44% better thermal comfort, and healthier CO2 concentrations over a one-year simulation, with reduced training time thanks to the integration of the behavior cloning learning technique.

show abstract

A review of reinforcement learning methodologies for controlling occupant comfort in buildings

Cited by 119 publications

References 59 publications

Reinforcement learning for building controls: The opportunities and challenges

Reinforcement learning for building controls: The opportunities and challenges

Deep Reinforcement Learning for Smart Home Energy Management

Reinforcement Learning-Based School Energy Management System

Contact Info

Product

Resources

About