Gigawatt-hour scale savings on a budget of zero: Deep reinforcement learning based optimal control of hot water systems

Abstract

Energy consumption for hot water production is a major draw in high efficiency buildings. Optimizing this has typically been approached from a thermodynamics perspective, decoupled from occupant influence. Furthermore, optimization usually presupposes existence of a detailed dynamics model for the hot water system. These assumptions lead to suboptimal energy efficiency in the real world. In this paper, we present a novel reinforcement learning based methodology which optimizes hot water production. The proposed methodology is completely generalizable, and does not require an offline step or human domain knowledge to build a model for the hot water vessel or the heating element. Occupant preferences too are learnt on the fly. The proposed system is applied to a set of 32 houses in the Netherlands where it reduces energy consumption for hot water production by roughly 20% with no loss of occupant comfort. Extrapolating, this translates to absolute savings of roughly 200 kWh for a single household on an annual basis. This performance can be replicated to any domestic hot water system and optimization objective, given that the fairly minimal requirements on sensor data are met. With millions of hot water systems operational worldwide, the proposed framework has the potential to reduce energy consumption in existing and new systems on a multi Gigawatt-hour scale in the years to come.

Predictive control based on occupant behavior prediction for domestic hot water system using data mining algorithm

“…By combining Equation (22)(23)(24), temperature at the inside top of the tank T t,top can be solved:…”

“…22 Occupant behavior has a significant impact on the results of future energy demand prediction and has become an important aspect in reducing thermal losses from energy inefficient buildings. 23 Kazmi et al 13 further indicated that obtaining a better understanding of occupant behavior only increases in importance as a necessary first step. However, for most data-driven predictive methods, a prediction is usually conducted on an aggregated energy consumption.…”

“…This is practically useful considering its application to not only new devices but also to the millions of devices that have already been installed in households. 23 A better balance between supply and demand will help decrease the remaining hot water in the tank and therefore realize greater energy saving. 24 This study aims to reduce the energy consumption of a DHW system during an operation period using a predictive control method and to achieve a better balance between supply and demand.…”

Analysis and Adaptation of Q-Learning Algorithm to Expert Controls of a Solar Domestic Hot Water System

“…Furthermore, the RL approach allows, using past experience and actual exploration, to account for the non-stationary features of the physical environment related to seasonal changes and natural degradation. An example application of RL techniques can be found in Reference [51], where the authors present a model-based RL algorithm to optimize the energy efficiency of hot water production systems. The authors make use of an ensemble of deep neural networks to approximate the transition function and get estimates of the current state uncertainty.…”

Trajectory Tracking With an Aggregation of Domestic Hot Water Heaters: Combining Model-Based and Model-Free Control in a Commercial Deployment

“…This challenge arises when controlling a large cluster of residential flexibility assets, for example, domestic hot water heaters (DHWHs). These loads are a high potential source of flexibility for demand response programs [2], [5], driven by their their decentralized abundance [6], considerable and efficient storage capacity [7] and its negligible inertia. Algorithms that tap into this resource must take into account a variety of factors, including: (1) intertemporal energy constraints, (2) the intrinsic uncertain user behavior, (3) methods to update models and control strategies as new information is collected and (4) the computational challenges associated with managing thousands to millions of devices to perform system-scale services.…”