Concentration bounds for temporal difference learning with linear function approximation: the case of batch data and uniform sampling

Abstract

We propose a stochastic approximation (SA) based method with randomization of samples for policy evaluation using the least squares temporal difference (LSTD) algorithm. Our proposed scheme is equivalent to running regular temporal difference learning with linear function approximation, albeit with samples picked uniformly from a given dataset. Our method results in an O(d) improvement in complexity in comparison to LSTD, where d is the dimension of the data. We provide non-asymptotic bounds for our proposed method, both in high probability and in expectation, under the assumption that the matrix underlying the LSTD solution is positive definite. The latter assumption can be easily satisfied for the pathwise LSTD variant proposed by Lazaric (J Mach Learn Res 13:3041-3074, 2012). Moreover, we also establish that using our method in place of LSTD does not impact the rate of convergence of the approximate value function to the true value function. These rate results coupled with the low computational complexity of our method make it attractive for implementation in big data settings, where d is large. A similar low-complexity alternative for least squares regression is well-known as the stochastic gradient descent (SGD) algorithm. We provide finite-time bounds for SGD. We demonstrate the practicality of our method as an efficient alternative for pathwise LSTD empirically by combining it with the least squares policy iteration algorithm in a traffic signal control application. We also conduct another set of experiments that combines the SA-based low-complexity variant for least squares regression with the LinUCB algorithm for contextual bandits, using the large scale news recommendation dataset from Yahoo.

N-SVRG: Stochastic Variance Reduction Gradient with Noise Reduction Ability for Small Batch Samples

“…Therefore, it is better to use the random variable Z as an unbiased estimate of μ than X. From the formula, it can be seen that the variance of Z is sufficiently small as long as the random variable X is guaranteed to show a certain correlation with Y [3], so Y is also called the control variable of X. This is the control variable method.…”

“…For SGD variance problem, there are three mainstream methods to reduce the variance of sampling at present that include importance sampling, hierarchical sampling method and control variable method. The objective function in machine learning is usually solved using the Batch Gradient Descent (BGD) or SGD [3]. BGD algorithm computes the gradients of all samples for each iteration to perform the weight update, and the latter randomly selects one training sample at a time to update the parameters by computing the sample gradients.…”

Online Estimation and Optimization of Utility-Based Shortfall Risk

“…) in (33) by using arguments similar to those used in arriving at Eq. (79) in [37]. In particular, the latter bound uses Jensen's inequality and the convexity of f (x) = x −2α exp(x 1−α ).…”

Finite time analysis of temporal difference learning with linear function approximation: Tail averaging and regularisation

“…Analysis of TD algorithms is challenging, and researchers have devoted significant effort in studying its asymptotic properties [7,11,15,19]. In recent years, there has been an interest in characterising the finite-time behaviour of TD, and several papers [1,2,3,9,13] have tackled this problem under various assumptions. For iterations/updates, most existing works either provide a 1 (with universal step-size) [1,3] or a 1 (with constant step size) [1,9,13] convergence rate to the TD-fixed point ★ defined as ★ −1 (see Section 2 for the notational information).…”