Experimental analysis of data-driven control for a building heating system

Abstract

Driven by the opportunity to harvest the flexibility related to building climate control for demand response applications, this work presents a data-driven control approach building upon recent advancements in reinforcement learning. More specifically, modelassisted batch reinforcement learning is applied to the setting of building climate control subjected to a dynamic pricing. The underlying sequential decision making problem is cast on a Markov decision problem, after which the control algorithm is detailed. In this work, fitted Q-iteration is used to construct a policy from a batch of experimental tuples. In those regions of the state space where the experimental sample density is low, virtual support tuples are added using an artificial neural network. Finally, the resulting policy is shaped using domain knowledge. The control approach has been evaluated quantitatively using a simulation and qualitatively in a living lab. From the quantitative analysis it has been found that the control approach converges in approximately 20 days to obtain a control policy with a performance within 90% of the mathematical optimum. The experimental analysis confirms that within 10 to 20 days sensible policies are obtained that can be used for different outside temperature regimes.

Convolutional Neural Networks for Automatic State-Time Feature Extraction in Reinforcement Learning Applied to Residential Load Control

“…the system dynamics are non-stationary, it is important to include at time-dependent state component to capture these patterns. As in [15], the time-dependent state component X t contains information related to timing. In this work, this component contains the hour of the day:…”

“…The backup controller guarantees the comfort settings of the end user by overruling the requested control action u i k when the comfort constraints of the end user are violated. For example, if the temperature of TCL i drops below T i k the backup controller will activate the TCL, independent of the requested control action, resulting in u phys,i k , which is needed to calculate the cost (15).…”

“…For example when using Q-learning, an established form of RL, a control policy is learned by estimating a state-action value function, or Q-function based upon interactions with the system to be controlled. In [13], [14], [15], RL has been been applied to the setting of residential demand response at the device level, whilst in [14] and [16], RL has been used in an aggregateand-dispatch setting with a large cluster of TCLs.…”

Smart buildings as Cyber-Physical Systems: Data-driven predictive control strategies for energy efficiency

“…As no thermal or physical modeling is involved, the approach is transferable to other buildings and other climatic zones without requiring extensive effort. [128] uses an ensemble of 40 NN to assist batch RL in creating an efficient HVAC DR controller able to control on-off decisions. A simulation of 40 days with different temperature regimes validates the approach.…”

Real‐time iterative learning control‐two applications with time scales between years and nanoseconds

“…The periodicity is not exactly the same from trial to trial, but if all previous trials are stored, then one can use the entire set of data of all previous trials and choose the one that fits best with the next trial. An iterative learning algorithm for the control of the building temperature with respect to the change of the ambient conditions from day to day was investigated in the work of Minakais et al 38 A data-driven control aproach for a building heating system was shown in the work of Costanzo et al 39 For this application, a simple linearized model of a heating system is used for the MPC. An additional data-driven iterative learning controller will be used to compensate for the modeling error and generate an optimal reference trajectory for the MPC.…”