Autonomous HVAC Control, A Reinforcement Learning Approach

Abstract

Abstract-Recent high profile developments of autonomous learning thermostats by companies such as Nest Labs and Honeywell have brought to the fore the possibility of ever greater numbers of intelligent devices permeating our homes and working environments into the future. However, the specific learning approaches and methodologies utilised by these devices have never been made public. In fact little information is known as to the specifics of how these devices operate and learn about their environments or the users who use them. This paper proposes a suitable learning architecture for such an intelligent thermostat in the hope that it will benefit further investigation by the research community. Our architecture comprises a number of different learning methods each of which contributes to create a complete autonomous thermostat capable of controlling a HVAC system. A novel state action space formalism is proposed to enable a Reinforcement Learning agent to successfully control the HVAC system by optimising both occupant comfort and energy costs. Our results show that the learning thermostat can achieve cost savings of 10% over a programmable thermostat, whilst maintaining high occupant comfort standards.

A Long-Short Term Memory Recurrent Neural Network Based Reinforcement Learning Controller for Office Heating Ventilation and Air Conditioning Systems

“…The various advantages of reinforcement learning methods have prompted some research directions [25][26][27] in applying it to HVAC control. For example, Gregor.…”

Towards plug&play smart thermostats inspired by reinforcement learning

“…RL is gaining attention nowadays and a growing use in the field of embedded systems is observed [28]. Several state-of-the-art approaches use RL for HVAC control [5,11,33]. Usual criticism to RL is the instability at the initial system period, as well as prolonged learning periods [1].…”

“…Regarding RL, an examination its application on Smart Thermostats has been introduced in [5]. In this work, energy cost corresponds to a reward of −1 when the HVAC is on but no actual energy costs are integrated.…”

Transfer Learning Applied to Reinforcement Learning-Based HVAC Control

“…However, its application can consume large amounts of energy if the management and control of the equipment is not operated in an optimal manner. A number of solutions, based on smart-thermostats, and learning user occupancy patterns have arisen recently and are replacing manual control configurations [1,12,13]. Prior to the introduction of adaptive control solutions, the most common method was the programmable schedule, which allowed the operator to choose the period to turn the HVAC system on/off given the most likely occupancy patterns and specify the temperature set-point of the given room.…”

“…Prior to the introduction of adaptive control solutions, the most common method was the programmable schedule, which allowed the operator to choose the period to turn the HVAC system on/off given the most likely occupancy patterns and specify the temperature set-point of the given room. This achieved energy and cost reductions when compared to leaving the HVAC always on, but recent work [1] has proven that machine learning-based adaptive methods can achieve even greater cost reductions whilst maintaining equivalent occupant comfort levels. Whilst achieving a sustainable solution with a better efficiency using machine learning (ML) techniques has demonstrated efficacy in the domain, one problem is the amount of time and data required to train effective policies in the absence of prior knowledge.…”