On the Convergence Rate of Incremental Aggregated Gradient Algorithms

Abstract

Motivated by applications to distributed optimization over networks and large-scale data processing in machine learning, we analyze the deterministic incremental aggregated gradient method for minimizing a finite sum of smooth functions where the sum is strongly convex. This method processes the functions one at a time in a deterministic order and incorporates a memory of previous gradient values to accelerate convergence. Empirically it performs well in practice; however, no theoretical analysis with explicit rate results was previously given in the literature to our knowledge, in particular most of the recent efforts concentrated on the randomized versions. In this paper, we show that this deterministic algorithm has global linear convergence and characterize the convergence rate. We also consider an aggregated method with momentum and demonstrate its linear convergence. Our proofs rely on a careful choice of a Lyapunov function that offers insight into the algorithm's behavior and simplifies the proofs considerably

Global convergence rate of incremental aggregated gradient methods for nonsmooth problems

“…In particular, we show that in order to achieve an ǫ-optimal solution, the PIAG algorithm requires O(QK 2 log 2 (QK) log(1/ǫ)) iterations, or equivalently O(QK 2 log(1/ǫ)) iterations, where the tilde is used to hide the logarithmic terms in Q and K. This result improves upon the condition number dependence of the deterministic IAG for smooth problems [12], where the authors proved that to achieve an ǫ-optimal solution, the IAG algorithm requires O(Q 2 K 2 log(1/ǫ)) iterations. We also note that two recent independent papers [9,15] have analyzed the convergence rate of the prox-gradient algorithm (which is a special case of our algorithm with K = 0, i.e., where we have access to a full gradient at each iteration instead of an aggregated gradient) under strong convexity type assumptions and provided linear rate estimates.…”

“…This is in contrast with the recent analysis of the IAG algorithm provided in [12], which used distances of the iterates to the optimal solution as a Lyapunov function and relied on the smoothness of the problem to bound the gradient errors with distances. This approach does not extend to the non-smooth composite case, which motivates a new analysis using function values and the properties of the proximal operator.…”

“…The IG method processes the component functions one at a time by taking steps along the gradient of each individual function in a sequential manner, following a cyclic order [26,27] or a randomized order [13,22,27]. A particular randomized order, which at each iteration independently picks a component function uniformly at random from all component functions leads to the popular stochastic gradient descent (SGD) method.…”

“…Blatt et al showed that under some assumptions, for a sufficiently small constant step size, the IAG method is globally convergent and when the component functions are quadratics, it achieves a linear rate. Two recent papers, [23] and [12], investigated the convergence rate of this method for general component functions that are convex and smooth (i.e., with Lipschitz gradients), where the sum of the component functions is strongly convex: In [23], the authors focused on a randomized version, called stochastic average gradient (SAG) method (which samples the component functions independently similar to SGD), and showed that it achieves a linear rate using a proof that relies on the stochastic nature of the algorithm. In a more recent work [12], the authors focused on deterministic IAG (i.e., component functions processed using an arbitrary deterministic order) and provided a simple analysis that uses a delayed dynamical system approach to study the evolution of the iterates generated by this algorithm.…”

“…For instance, if the functions are processed in a cyclic order, we have K = m − 1 [13,26]. On the other hand, K = 0 corresponds to the case where we have the full gradient of the function f (x) at each iteration (i.e., g k = ∇f (x k )) and small K may represent a setting in which the gradients of the component functions are sent to a processor with some delay upper bounded by K.…”