Learning continuous $Q$-functions using generalized Benders cuts

Abstract

Q-functions are widely used in discrete-time learning and control to model future costs arising from a given control policy, when the initial state and input are given. Although some of their properties are understood, Q-functions generating optimal policies for continuous problems are usually hard to compute. Even when a system model is available, optimal control is generally difficult to achieve except in rare cases where an analytical solution happens to exist, or an explicit exact solution can be computed. It is typically necessary to discretize the state and action spaces, or parameterize the Q-function with a basis that can be hard to select a priori. This paper describes a model-based algorithm based on generalized Benders theory that yields ever-tighter outer-approximations of the optimal Qfunction. Under a strong duality assumption, we prove that the algorithm yields an arbitrarily small Bellman optimality error at any finite number of arbitrary points in the stateinput space, in finite iterations. Under additional assumptions, the same guarantee holds when the inputs are determined online by the algorithm's updating Q-function. We demonstrate these properties numerically on scalar and 8-dimensional systems.

Learning Q-Function Approximations for Hybrid Control Problems

“…This results in an approximation which lower bounds Q (N ) everywhere. Following [9], [13] we choose the cuts to be linearly separable in their arguments:…”

“…The proof is an adaptation of the proof of Lemma III.1 in [9] that uses the properties of the dual variables to prove the global lower bounding property for a cut added at any [13] carry over to the present setting and describe the conditions under which new cuts improve the approximation and by how much. We highlight that the addition of the new cut q I+1 at the parameter {s}…”

Learning $Q$-function approximations for hybrid control problems

“…The proof is a simple adaptation of the proof of Lemma III.1 in [7]. Lemmas III.2-4 of [10] carry over to the present setting but are omitted in the interest of space. We highlight that the addition of the new cut q I+1 at the parameter {s} N -1 0 ∈ S (N ) (x 0 ) leads to an improvement of…”

“…This results in an approximation which lower bounds Q (N ) ⋆ everywhere. Following [7], [10] we choose the cuts to be linearly separable in their arguments:…”