Acceleration of Stochastic Approximation by Averaging

Abstract

A new recursive algorithm of stochastic approximation type with the averaging of trajectories is investigated. Convergence with probability one is proved for a variety of classical optimization and identification problems. It is also demonstrated for these problems that the proposed algorithm achieves the highest possible rate of convergence.

LMS-2: Towards an algorithm that is as cheap as LMS and almost as efficient as RLS

“…Their analysis shows that the performance of one-timescale algorithms can be enhanced (via the asymptotic rate of convergence) using the above iterate averaging. Iterate averaging such as suggested by [10] can in fact be easily obtained using two timescales. Konda and Tsitsiklis [7] follow this approach and obtain rate of convergence for general two-timescale algorithms in the case of linear systems under certain assumptions.…”

“…Konda and Tsitsiklis [7] follow this approach and obtain rate of convergence for general two-timescale algorithms in the case of linear systems under certain assumptions. They generalize the iterate averaging idea of [10] by considering more general (though linear) recursions. In the recursions in [7] along the faster timescale, one does not necessarily do iterate averaging as in [10] but these iterations can be more general.…”

“…They generalize the iterate averaging idea of [10] by considering more general (though linear) recursions. In the recursions in [7] along the faster timescale, one does not necessarily do iterate averaging as in [10] but these iterations can be more general. (Our two-timescale algorithm is also for a linear system of equations.)…”

“…Again, the origin is a unique globally asymptotically stable equilibrium for (10). Now for the ODE (9), θ = Φ −1 E [yφ] is the unique globally asymptotically stable equilibrium.…”

“…where β t = 1/t, then the iterates θ t can be shown to converge to θ at an asymptotically optimal rate (see, [10] and Chapter 11 of [8]). This idea works more generally and is called iterate-averaging and was developed independently by Polyak [9] and Ruppert [11].…”