Optimal Rate of Convergence for Quasi-Stochastic Approximation

Abstract

The Robbins-Monro stochastic approximation algorithm is a foundation of many algorithmic frameworks for reinforcement learning (RL), and often an efficient approach to solving (or approximating the solution to) complex optimal control problems. However, in many cases practitioners are unable to apply these techniques because of an inherent high variance. This paper aims to provide a general foundation for "quasistochastic approximation," in which all of the processes under consideration are deterministic, much like quasi-Monte-Carlo for variance reduction in simulation. The variance reduction can be substantial, subject to tuning of pertinent parameters in the algorithm. This paper introduces a new coupling argument to establish optimal rate of convergence provided the gain is sufficiently large. These results are established for linear models, and tested also in non-ideal settings.A major application of these general results is a new class of RL algorithms for deterministic state space models. In this setting, the main contribution is a class of algorithms for approximating the value function for a given policy, using a different policy designed to introduce exploration.

Model-Free Primal-Dual Methods for Network Optimization with Application to Real-Time Optimal Power Flow

“…While the stylized algorithm (4) and approximation ( 6) is used here for an illustration of the main ideas, the paper considers a more general convex constrained optimization framework, and develops model-free primal-dual methods to track the optimal trajectories. Using a deterministic exploration approach reminiscent to the quasi-stochastic approximation method [5], this paper provides design principles for the exploration signal ξ, as well as other algorithmic parameters, to ensure stability and tracking guarantees. In particular, we show that under some conditions, the iterates x (k) converge within a ball around the optimal solution of (2).…”

“…The paper then develops distributed algorithms based on the zero-order approximation of the method of multipliers. In contrast to our paper, [21] considers a stochastic exploration signal for the gradient estimation, and typically requires N > 2 function evaluations to reduce the estimation variance [21, Lemma 1]; see also [5] for the detailed analysis of the advantage of deterministic vs stochastic exploration. Moreover, it considers a static optimization problem.…”

“…Recall that the exploration signal ξ (k) is sampled from a continuous-time signal ξ(t). Recent work on quasi-stochastic approximation [5], [33] and extremum seeking control [1] has shown that certain deterministic exploration signals can help reduce the asymptotic variance and improve convergence, in comparison to random exploration (see Fig. 1 of [33]).…”

“…in which Σ ξ is positive definite, and the 1/T bound for the error term is independent of t (this is the assumption used in [5]). An example is the signal defined in (16) in which the frequencies may not have a common multiple.…”

“…We first show that the primal step ( 18) is approximately equivalent to an averaged primal step, where the singular "gain matrix" ξξ T is replaced with the identity matrix (20). This analysis is performed for the algorithm defined in continuous time, which is justified using standard ODE approximation techniques from the stochastic approximation literature [5], [11] or more recent literature on optimization [32], [37]. We then apply the results of [8] to the resulting approximate primal-dual algorithm to show tracking of the desired trajectory {z ( * ,k) } defined by (11).…”