On the rates of convergence of parallelized averaged stochastic gradient algorithms

Abstract

The growing interest for high dimensional and functional data analysis led in the last decade to an important research developing a consequent amount of techniques. Parallelized algorithms, which consist in distributing and treat the data into different machines, for example, are a good answer to deal with large samples taking values in high dimensional spaces. We introduce here a parallelized averaged stochastic gradient algorithm, which enables to treat efficiently and recursively the data, and so, without taking care if the distribution of the data into the machines is uniform. The rate of convergence in quadratic mean as well as the asymptotic normality of the parallelized estimates are given, for strongly and locally strongly convex objectives.

An Accelerated Second-Order Method for Distributed Stochastic Optimization

“…Indeed, if m √ N the minimum in the exponent in (8) is achieved on the first term, and no improvement in the bound is guaranteed. Moreover, the similar limit m ≃ √ N can be obtained via SA-based method from [17] with the guarantee (6).…”

“…But it is not known whether it is accurate or not, since there is no method on which it reached. In the paper [17] authors get a better convergence rate considering non-accelerated parallelized SGD with specific step size and only one communication at the end, assuming stronger local smoothness of objective near the solution…”

“…The authors of [17], under an additional assumption of stronger local smoothness of f around x * , propose an algorithm with only one round of communication (T = 1) with the following guarantee…”

Non-asymptotic analysis of Stochastic approximation algorithms for streaming data

“…k and which can be so recursively invert with the help of Riccati/Shermann-Morrisson's formula (see Bercu et al (2020); Boyer and Godichon-Baggioni (2020); Godichon-Baggioni et al (2022)). In order to verify (H1b), one can consider the following version of the estimate of the Hessian…”

“…k where e k is the k-th (modulo d) canonical vector (see Bercu et al (2021); Godichon-Baggioni et al (2022)). We can now obtain a first rate of convergence of the estimates.…”