Convergence Rate of Incremental Subgradient Algorithms

Nedić, Angelia; Bertsekas, Dimitri P.

doi:10.1007/978-1-4757-6594-6_11

Cited by 162 publications

(213 citation statements)

References 16 publications

Supporting

Mentioning

206

Contrasting

Order By: Relevance

“…However, this unfortunately leads to the same slow O(1/ √ ) and O(1/ ) convergence rates of the sub-gradient method. But interestingly, if we still use a constant step-size at each iteration for the stochastic gradient method the algorithm is known to quickly reduce the initial error, even if it has a nonvanishing optimization error [24]. We have observed this for the stochastic gradient descent example in Figure 1.…”

Section: Algorithmmentioning

confidence: 80%

“…Similar to the coordinate descent methods, the crucial design problem in the stochastic gradient methods is the selection of the data points j at each iteration. Analogously, we obtain better convergence rates by choosing the j uniformly at random rather than cycling through the data [24].…”

Section: Algorithmmentioning

confidence: 97%

“…We have observed this for the stochastic gradient descent example in Figure 1. Indeed, while stochastic gradient descent methods have historically been notoriously hard to tune, recent results show that using large step sizes and weighted averaging of the iterates (cf., Figure 1) allows us to achieve optimal convergence rates while being robust to the setting of the step size and the modeling assumptions [24,25]. For example, recent work has shown [25] that an averaged stochastic gradient iteration with a constant step-size achieves an O(1/ε) convergence rate even without strong convexity under joint self-concordance-like and Lipschitz gradient assumptions.…”

Section: Algorithmmentioning

confidence: 99%

See 2 more Smart Citations

Convex Optimization for Big Data: Scalable, randomized, and parallel algorithms for big data analytics

Cevher

Becker

Schmidt

2014

IEEE Signal Process. Mag.

275

255

View full text Add to dashboard Cite

This article reviews recent advances in convex optimization algorithms for Big Data, which aim to reduce the computational, storage, and communications bottlenecks. We provide an overview of this emerging field, describe contemporary approximation techniques like first-order methods and randomization for scalability, and survey the important role of parallel and distributed computation.The new Big Data algorithms are based on surprisingly simple principles and attain staggering accelerations even on classical problems. Convex optimization in the wake of Big DataConvexity in signal processing dates back to the dawn of the field, with problems like least-squares being ubiquitous across nearly all sub-areas. However, the importance of convex formulations and optimization has increased even more dramatically in the last decade due to the rise of new theory for structured sparsity and rank minimization, and successful statistical learning models like support vector machines. These formulations are now employed in a wide variety of signal processing applications including compressive sensing, medical imaging, geophysics, and bioinformatics [1-4].There are several important reasons for this explosion of interest, with two of the most obvious ones being the existence of efficient algorithms for computing globally optimal solutions and the ability to use convex geometry to prove useful properties about the solution [1,2]. A unified convex formulation also transfers useful knowledge across different disciplines, such as sampling and computation, that focus on different aspects of the same underlying mathematical problem [5].However, the renewed popularity of convex optimization places convex algorithms under tremendous pressure to accommodate increasingly large data sets and to solve problems in unprecedented dimensions. Internet, text, and imaging problems (among a myriad of other examples) no longer produce data sizes from megabytes to gigabytes, but rather from terabytes to exabytes. Despite

show abstract

Section: Algorithmmentioning

confidence: 80%

Section: Algorithmmentioning

confidence: 97%

Section: Algorithmmentioning

confidence: 99%

See 1 more Smart Citation

Convex Optimization for Big Data: Scalable, randomized, and parallel algorithms for big data analytics

Cevher

Becker

Schmidt

2014

IEEE Signal Process. Mag.

275

255

View full text Add to dashboard Cite

show abstract

“…Without the presence of noise, the incremental method has been studied by Kibardin [14], Nedić and Bertsekas [18,19], Nedić, Bertsekas, and Borkar [17] (see also Nedić [20]), Ben-Tal, Margalit, and Nemirovski [1], and Kiwiel [15]. The incremental idea have been extended to min-max problems through the use of bundle methods by Gaudioso, Giallombardo, and Miglionico [11].…”

Section: Implications For Incremental K -Subgradient Methodsmentioning

confidence: 99%

The effect of deterministic noise in subgradient methods

Nedić

Bertsekas

2009

Math. Program.

Self Cite

View full text Add to dashboard Cite

In this paper, we study the influence of noise on subgradient methods for convex constrained optimization. The noise may be due to various sources, and is manifested in inexact computation of the subgradients and function values. Assuming that the noise is deterministic and bounded, we discuss the convergence properties for two cases: the case where the constraint set is compact, and the case where this set need not be compact but the objective function has a sharp set of minima (for example the function is polyhedral). In both cases, using several different stepsize rules, we prove convergence to the optimal value within some tolerance that is given explicitly in terms of the errors. In the first case, the tolerance is nonzero, but in the second case, the optimal value can be obtained exactly, provided the size of the error in the subgradient computation is below some threshold. We then extend these results to objective functions that are the sum of a large number of convex functions, in which case an incremental subgradient method can be used.

show abstract

“…However, it has a relatively weak convergence rate of O( √ ε) [109], i.e., reducing the distance between w cur and w * by a factor of ε can require O( 1 ε 2 ) iterations. Given that each iteration requires multiple costly steps of loss-augmented prediction, it is natural to look for efficient way to optimize Equation (6.2).…”

Section: Subgradient Descent Minimizationmentioning

confidence: 99%