Minimizing Finite Sums with the Stochastic Average Gradient

Schmidt, Mark; Roux, Nicolas Le; Bach, Francis

doi:10.48550/arxiv.1309.2388

Cited by 91 publications

(171 citation statements)

References 0 publications

Supporting

Mentioning

165

Contrasting

Order By: Relevance

“…V C), as opposed to our assumption (A4) where the gradients are assumed to be known exactly. To account for noisy gradients, it is possible to use convergence bounds of stochastic gradient descent [47,58] to estimate a bound on the number of gradient descent steps. Second-order optimization methods such as stochastic reconfiguration/natural gradient [59,60] can potentially show a significant advantage over first-order optimization methods, in terms of scaling with the minimum gap of the time-dependent Hamiltonian Ĥ(t).…”

Section: (B8)mentioning

confidence: 99%

Variational neural annealing

et al. 2021

View full text Add to dashboard Cite

Many important challenges in science and technology can be cast as optimization problems. When viewed in a statistical physics framework, these can be tackled by simulated annealing, where a gradual cooling procedure helps search for groundstate solutions of a target Hamiltonian. While powerful, simulated annealing is known to have prohibitively slow sampling dynamics when the optimization landscape is rough or glassy. Here we show that by generalizing the target distribution with a parameterized model, an analogous annealing framework based on the variational principle can be used to search for groundstate solutions. Modern autoregressive models such as recurrent neural networks provide ideal parameterizations since they can be exactly sampled without slow dynamics even when the model encodes a rough landscape. We implement this procedure in the classical and quantum settings on several prototypical spin glass Hamiltonians, and find that it significantly outperforms traditional simulated annealing in the asymptotic limit, illustrating the potential power of this yet unexplored route to optimization.

show abstract

Section: (B8)mentioning

confidence: 99%

Variational neural annealing

et al. 2021

View full text Add to dashboard Cite

show abstract

“…We used a variant introduced in [39], which uses Barzilai-Borwein in order to adapt the step-size, since gradient-Lipschitz constants are unavailable in the considered setting. We consider this version of variance reduction, since alternatives such as SAGA [13] and SAG [35] do not propose variants with Barzilai-Borwein type of step-size selection.…”

Section: Methodsmentioning

confidence: 99%

“…In the recent years, much effort has been made to minimize strongly convex finite sums with first order information. Recent developments, combining both numerical efficiency and sound theoretical guarantees, such as linear convergence rates, include SVRG [19], SAG [35], SDCA [36] or SAGA [13] to solve the following problem:…”

Section: Introductionmentioning

confidence: 99%

“…Standard first-order batch solvers (non stochastic) for composite convex objectives are ISTA and its accelerated version FISTA [5] and first-order stochastic solvers are mostly built on the idea of Stochastic Gradient Descent (SGD) [34]. Recently, stochastic solvers based on a combination of SGD and the Monte-Carlo technique of variance reduction [35], [36], [19], [13] turn out to be both very efficient numerically (each update has a complexity comparable to vanilla SGD) and very sound theoretically, because of strong linear convergence guarantees, that match or even improve the one of batch solvers. These algorithms involve gradient steps on the smooth part of the objective and theoretical guarantees justify such steps under the gradient-Lipschitz assumptions thanks to the descent lemma [7,Proposition A.24].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Dual optimization for convex constrained objectives without the gradient-Lipschitz assumption

Bompaire¹,

Bacry²,

Gaı̈ffas³

2018

Preprint

View full text Add to dashboard Cite

The minimization of convex objectives coming from linear supervised learning problems, such as penalized generalized linear models, can be formulated as finite sums of convex functions. For such problems, a large set of stochastic first-order solvers based on the idea of variance reduction are available and combine both computational efficiency and sound theoretical guarantees (linear convergence rates) [19], [35], [36], [13]. Such rates are obtained under both gradient-Lipschitz and strong convexity assumptions. Motivated by learning problems that do not meet the gradient-Lipschitz assumption, such as linear Poisson regression, we work under another smoothness assumption, and obtain a linear convergence rate for a shifted version of Stochastic Dual Coordinate Ascent (SDCA) [36] that improves the current state-of-the-art. Our motivation for considering a solver working on the Fenchel-dual problem comes from the fact that such objectives include many linear constraints, that are easier to deal with in the dual. Our approach and theoretical findings are validated on several datasets, for Poisson regression and another objective coming from the negative log-likelihood of the Hawkes process, which is a family of models which proves extremely useful for the modeling of information propagation in social networks and causality inference [12], [14].

show abstract

“…The random variable Y is "realized" either by drawing an observation from an existing dataset or by using Monte Carlo. Stochastic Approximation (SA) [13], also known as Stochastic Gradient Descent (SGD), and its variants [5,7,8,11,12,14], form the popular class of methods that are used for solving problems of the type (Q). Over the past few years, there has been an increased interest in stochastic quasi-Newton methods [1,4,6,9,10,[15][16][17] that incorporate curvature information within stochastic gradient based methods.…”

Section: Introductionmentioning

confidence: 99%

Retrospective Approximation for Smooth Stochastic Optimization

Newton¹,

Bollapragada²,

Pasupathy³

et al. 2021

Preprint

View full text Add to dashboard Cite

We consider stochastic optimization problems where a smooth (and potentially nonconvex) objective is to be minimized using a stochastic first-order oracle. These type of problems arise in many settings from simulation optimization to deep learning. We present Retrospective Approximation (RA) as a universal sequential sample-average approximation (SAA) paradigm where during each iteration k, a sample-path approximation problem is implicitly generated using an adapted sample size M k , and solved (with prior solutions as "warm start") to an adapted error tolerance k , using a "deterministic method" such as the line search quasi-Newton method. The principal advantage of RA is that decouples optimization from stochastic approximation, allowing the direct adoption of existing deterministic algorithms without modification, thus mitigating the need to redesign algorithms for the stochastic context. A second advantage is the obvious manner in which RA lends itself to parallelization. We identify conditions on {M k , k ≥ 1} and { k , k ≥ 1} that ensure almost sure convergence and convergence in L 1 -norm, along with optimal iteration and work complexity rates. We illustrate the performance of RA with line-search quasi-Newton on an ill-conditioned least squares problem, as well as an image classification problem using a deep convolutional neural net.

show abstract

Minimizing Finite Sums with the Stochastic Average Gradient

Cited by 91 publications

References 0 publications

Variational neural annealing

Variational neural annealing

Dual optimization for convex constrained objectives without the gradient-Lipschitz assumption

Retrospective Approximation for Smooth Stochastic Optimization

Contact Info

Product

Resources

About