Second-Order Stochastic Optimization for Machine Learning in Linear Time

Agarwal, Naman; Bullins, Brian; Hazan, Elad

doi:10.48550/arxiv.1602.03943

Cited by 19 publications

(57 citation statements)

References 9 publications

(13 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This provides evidence to the necessity of adaptive sampling schemes, and a dimension-dependent analysis, which indeed accords with some recently proposed algorithms and derivations, e.g. Agarwal et al [2016], Xu et al [2016]. We note that the limitations arising from oblivious optimization schemes (in a somewhat stronger sense) was also explored in Arjevani and Shamir [2016a,b].…”

Section: Introductionsupporting

confidence: 84%

“…Therefore, we conjecture that this approach cannot lead to better worst-case results. Agarwal et al [2016] develop another line of stochastic second-order methods, which are based on the observation that the Newton step (∇ 2 F (w)) −1 ∇F (w) is the solution of the system of linear equations…”

Section: Comparison To Existing Approachesmentioning

confidence: 99%

“…Building on this, several recent papers proposed and analyzed second-order methods for finite-sum problems, which utilize Hessians of the individual functions f i (see for instance Erdogdu and Montanari [2015], Agarwal et al [2016], Pilanci and Wainwright [2015], Roosta-Khorasani and Mahoney [2016a,b], Bollapragada et al [2016], Xu et al [2016] and references therein). These can all be viewed as approximate Newton methods, which replace the actual Hessian ∇ 2 F (w) = 1 n n i=1 ∇ 2 f i (w) in Eq.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Oracle complexity of second-order methods for smooth convex optimization

2018

View full text Add to dashboard Cite

Second-order methods, which utilize gradients as well as Hessians to optimize a given function, are of major importance in mathematical optimization. In this work, we prove tight bounds on the oracle complexity of such methods for smooth convex functions, or equivalently, the worst-case number of iterations required to optimize such functions to a given accuracy. In particular, these bounds indicate when such methods can or cannot improve on gradient-based methods, whose oracle complexity is much better understood. We also provide generalizations of our results to higher-order methods.• Perhaps unexpectedly, Eq. (5) establishes that one cannot avoid in general a polynomial dependence on geometry-dependent "condition numbers" of the form µ 1 /λ or µ 2 D/λ, even with second-order methods. This is despite the ability of such methods to favorably alter the geometry of the problem (for example, the Newton method is well-known to be affine invariant).• To improve on the oracle complexity of first-order methods for strongly-convex problems (Eq. (3)) by more than logarithmic factors, one cannot avoid a polynomial dependence on the initial distance D to the optimum. This is despite the fact that the dependence on D with first-order methods is only logarithmic. In fact, when D is sufficiently large (of order µ 7/4 1 µ 2 λ 3/4 or larger), second-order methods cannot improve on the oracle complexity of first-order methods by more than logarithmic factors. 1 Assuming f is twice-differentiable, this corresponds to ∇ 2 f (w) λI uniformly for all w.

show abstract

Section: Introductionsupporting

confidence: 84%

Section: Comparison To Existing Approachesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Oracle complexity of second-order methods for smooth convex optimization

2018

View full text Add to dashboard Cite

show abstract

“…Fortunately, we do not need to calculate the full Hessian, only Hessian-vector products [e.g., H −1 θ ∇L r in Eqs. (1) and (3)] or the top part of the Hessian spectrum, which significantly reduces the computational complexity of the problem [65]. The inverse of the Hessian for influence functions, RUE, and RelatIF can be approximated with so-called stochastic approximation with LiSSA [65].…”

Section: F Practical Aspects Of the Hessian Computationmentioning

confidence: 99%

“…(1) and (3)] or the top part of the Hessian spectrum, which significantly reduces the computational complexity of the problem [65]. The inverse of the Hessian for influence functions, RUE, and RelatIF can be approximated with so-called stochastic approximation with LiSSA [65]. Additionally, the authors of RelatIF approximated the normalization factor in Eq.…”

Section: F Practical Aspects Of the Hessian Computationmentioning

confidence: 99%

Hessian-based toolbox for reliable and interpretable machine learning in physics

Dawid,

Huembeli,

Tomza

et al. 2021

Preprint

View full text Add to dashboard Cite

Optimization Methods for Inverse Problems

Roosta-Khorasani

Cui

2019

MATRIX Book Series

View full text Add to dashboard Cite

Optimization plays an important role in solving many inverse problems. Indeed, the task of inversion often either involves or is fully cast as a solution of an optimization problem. In this light, the mere non-linear, non-convex, and large-scale nature of many of these inversions gives rise to some very challenging optimization problems. The inverse problem community has long been developing various techniques for solving such optimization tasks. However, other, seemingly disjoint communities, such as that of machine learning, have developed, almost in parallel, interesting alternative methods which might have stayed under the radar of the inverse problem community. In this survey, we aim to change that. In doing so, we first discuss current state-of-the-art optimization methods widely used in inverse problems. We then survey recent related advances in addressing similar challenges in problems faced by the machine learning community, and discuss their potential advantages for solving inverse problems. By highlighting the similarities among the optimization challenges faced by the inverse problem and the machine learning communities, we hope that this survey can serve as a bridge in bringing together these two communities and encourage cross fertilization of ideas.

show abstract

Second-Order Stochastic Optimization for Machine Learning in Linear Time

Cited by 19 publications

References 9 publications

Oracle complexity of second-order methods for smooth convex optimization

Oracle complexity of second-order methods for smooth convex optimization

Hessian-based toolbox for reliable and interpretable machine learning in physics

Optimization Methods for Inverse Problems

Contact Info

Product

Resources

About