On the Taylor Expansion of Value Functions

Abstract

We introduce a framework for approximate dynamic programming that we apply to discrete time chains on Z d + with countable action sets. Our approach is grounded in the approximation of the (controlled) chain's generator by that of another Markov process. In simple terms, our approach stipulates applying a secondorder Taylor expansion to the value function to replace the Bellman equation with one in continuous space and time where the transition matrix is reduced to its first and second moments. In some cases, the resulting equation (which we label TCP) can be interpreted as corresponding to a Brownian control problem.When tractable, the TCP serves as a useful modeling tool. More generally, the TCP is a starting point for approximation algorithms. We develop bounds on the optimality gap-the sub-optimality introduced by using the control produced by the "Taylored" equation. These bounds can be viewed as a conceptual underpinning, analytical rather than relying on weak convergence arguments, for the good performance of controls derived from Brownian control problems. We prove that, under suitable conditions and for suitably "large" initial states, (i) the optimality gap is smaller than a 1 − α fraction of the optimal value, where α ∈ (0, 1) is the discount factor, and (ii) the gap can be further expressed as the infinite horizon discounted value with a "lower-order" per period reward. Computationally, our framework leads to an "aggregation" approach with performance guarantees. While the guarantees are grounded in PDE theory, the practical use of this approach requires no knowledge of that theory.

Kernel-based Diffusion Approximated Markov Decision Processes for Off-Road Autonomous Navigation and Control

“…We subtract both hand-sides by v π psq from Eq. ( 2) and then take Taylor expansions of value function around s up to second order [67]:…”

Open Problem—Regarding Static Priority Scheduling for Many-Server Queues with Reneging

“…When customer interarrival, service, and patience times are all exponentially distributed, the scheduling control problem can be written as a Markov decision problem, in which case we also want to bound the performance of threshold scheduling policies. In this setting, we are hopeful the methodology in Braverman et al (2019) could be helpful. (We note that a similar question holds in the single-server setting, which could be an easier place to begin.)…”

The Prelimit Generator Comparison Approach of Stein’s Method

“…Recent years have seen a growing use of the generator comparison approach of Stein's method to establish rates of convergence for steady-state diffusion approximations of Markov chains. One very active area has been the study of queueing and service systems; see, for example, Gurvich (2014a), Stolyar (2015), Braverman et al (2016Braverman et al ( , 2020Braverman et al ( , 2021, Ying (2016Ying ( , 2017, Braverman and Dai (2017), Dai and Shi (2017), Huang and Gurvich (2018), Feng and Shi (2018), Liu and Ying (2019), and Braverman (2020). In the typical setup, one considers a parametric family of continuous-time Markov chains (CTMCs) {X(t)} taking values in some discrete state space.…”