Approximate model-assisted Neural Fitted Q-Iteration

Abstract

Abstract-In this work, we propose an extension to the Neural Fitted Q-Iteration algorithm that utilizes a learned model to generate virtual trajectories which are used for updating the Q-function. Compared to standard NFQ, this combination has the potential to greatly reduce the amount of system interaction required to learn a good policy. At the same time, the approach still maintains the generalization ability of Q-learning. We provide a general formulation for approximate model-assisted fitted Q-learning, and examine the advantages of its neural implementation regarding interaction time and robustness. Its capabilities are illustrated with first results on a benchmark cart-pole regulation task, on which our method turns out to provide more general policies using much less interaction time.

Model-free control of thermostatically controlled loads connected to a district heating network

“…This domain knowledge can be incorporated through e.g. information regarding the shape of the policy [40] or through using a model similar as in [53]. A second point of…”

“…In [42] the authors conclude that especially for non-linear control problems, FQI can be a valuable alternative to MPC approaches with the extra advantage that FQI is a blind technique. Moreover, FQI and MPC can strengthen each other [42,53]. Although several BRL techniques have been proposed in the literature [36,52,44], this work focuses on FQI using extremely randomized trees as regression algorithm [47].…”

Reinforcement Learning Applied to an Electric Water Heater: From Theory to Practice

“…For example, the authors of [29] present a model-based policy search method that learns a Gaussian process to model uncertainties. In addition, inspired by [30], the authors of [31] demonstrate how a model-assisted batch RL technique can be applied to control a building heating system.…”

Model-based reinforcement learning for output-feedback optimal control of a class of nonlinear systems

“…Several different MBRL approaches have been developed in the literature over the last few decades. Imaginary roll-outs, i.e., the use of a model as a proxy for the real world to evaluate temporal difference errors (referred to as Bellman errors (BEs) in this paper) are explored in results such as [22] and [23]. While the sample efficiency is of the policy learning algorithms is improved, the performance of the method in [22] decays rapidly with model mismatch, and the method in [23] relies on fitting neural networks to dynamics, which is typically data-intensive, nullifying the sample efficiency gain in the policy learning algorithm.…”