A Proximal Stochastic Gradient Method with Progressive Variance Reduction

Abstract

We consider the problem of minimizing the sum of two convex functions: one is the average of a large number of smooth component functions, and the other is a general convex function that admits a simple proximal mapping. We assume the whole objective function is strongly convex. Such problems often arise in machine learning, known as regularized empirical risk minimization. We propose and analyze a new proximal stochastic gradient method, which uses a multi-stage scheme to progressively reduce the variance of the stochastic gradient. While each iteration of this algorithm has similar cost as the classical stochastic gradient method (or incremental gradient method), we show that the expected objective value converges to the optimum at a geometric rate. The overall complexity of this method is much lower than both the proximal full gradient method and the standard proximal stochastic gradient method.

Asynchronous incremental block-coordinate descent

“…Remark 1: Algorithm 2 and its analysis is slightly different than the delayed incremental gradient method that we presented in [9]. In the notation of this paper, our earlier algorithm would have a convergence factor on the same form as (8) but [10], it follows that our guaranteed upper bound improves upon the one in [9], especially for applications where the component functions vary substantially in smoothness.…”

Online game props recommendation with real assessments

“…7 in base solution to the following Eq. 10 by appending with a proximal operator [22] to obtain the sparse solutions for the linear coefficients w j,l j of each relevant prop j as …”

Distributed semi-stochastic optimization with quantization refinement

“…We first restate some useful results from [10]. Further let F (x) and R(x) have convexity parameters µ F and µ R , respectively.…”

“…We now proceed to prove Theorem 1. For brevity, we omit some details that are identical to those in the proof of Theorem 3.1 in [10]. We have indicated these omissions below.…”

“…We follow the same reasoning as in the proof of Theorem 3.1 in [10] to obtain the following expression, which is conditioned on x (st−1) and takes expectation with respect to ,…”