Global convergence rate analysis of unconstrained optimization methods based on probabilistic models

Cartis, Coralia; Scheinberg, Katya

doi:10.1007/s10107-017-1137-4

Cited by 121 publications

(225 citation statements)

References 19 publications

Supporting

Mentioning

221

Contrasting

Unclassified

Order By: Relevance

“…It is based on an inner product test that ensures that the search direction in (1.4) is a descent direction with high probability. In contrast to the test studied in [6][7][8]14], which we call the norm test, and which controls both the direction and length of the gradient approximation and promotes search directions that are close to the true gradient, the inner product test places more emphasis on generating descent directions and allows more freedom in their length. The numerical results presented in Section 5 suggest that the inner product test is efficient in practice, but in order to establish a Q-linear convergence rate for strongly convex functions, we must reinforce it with an additional mechanism that prevents search directions from becoming nearly orthogonal to the true gradient ∇F (x k ).…”

Section: Introductionmentioning

confidence: 99%

Adaptive Sampling Strategies for Stochastic Optimization

Bollapragada¹,

Byrd²,

Nocedal³

2018

SIAM J. Optim.

110

View full text Add to dashboard Cite

In this paper, we propose a stochastic optimization method that adaptively controls the sample size used in the computation of gradient approximations. Unlike other variance reduction techniques that either require additional storage or the regular computation of full gradients, the proposed method reduces variance by increasing the sample size as needed. The decision to increase the sample size is governed by an inner product test that ensures that search directions are descent directions with high probability. We show that the inner product test improves upon the well known norm test, and can be used as a basis for an algorithm that is globally convergent on nonconvex functions and enjoys a global linear rate of convergence on strongly convex functions. Numerical experiments on logistic regression problems illustrate the performance of the algorithm.

show abstract

Section: Introductionmentioning

confidence: 99%

Adaptive Sampling Strategies for Stochastic Optimization

Bollapragada¹,

Byrd²,

Nocedal³

2018

SIAM J. Optim.

110

View full text Add to dashboard Cite

show abstract

“…We note here that if p f = 1 then Assumption 2.4(iii) is not needed and condition p g > 1/2 is sufficient for the convergence results. This case can be considered as an extension of results in [6]. Before concluding this section, we state a result showing the relationship between the variance assumption on the function values and the probability of inaccurate estimates.…”

Section: Random Gradient and Function Estimatesmentioning

confidence: 85%

“…Before delving into the convergence statement and proof, we state some lemmas similar to those derived in [2,6,7].…”

Section: Useful Resultsmentioning

confidence: 99%

“…(whereÕ hides the log factor of 1/(1 − p g )), then Assumption 2.4(i) is satisfied. While G k is not known when |S k | is chosen, one can design a simple loop by guessing the value of G k and increasing the number of samples until (2.4) is satisfied, this procedure is discussed in [6]. Similarly to satisfy Assumption 2.4(ii), it is sufficient to compute F 0…”

Section: 3mentioning

confidence: 99%

See 1 more Smart Citation

A Stochastic Line Search Method with Expected Complexity Analysis

Paquette¹,

Scheinberg²

2020

SIAM J. Optim.

View full text Add to dashboard Cite

For deterministic optimization, line-search methods augment algorithms by providing stability and improved efficiency. We adapt a classical backtracking Armijo line-search to the stochastic optimization setting. While traditional line-search relies on exact computations of the gradient and values of the objective function, our method assumes that these values are available up to some dynamically adjusted accuracy which holds with some sufficiently large, but fixed, probability. We show the expected number of iterations to reach a near stationary point matches the worst-case efficiency of typical first-order methods, while for convex and strongly convex objective, it achieves rates of deterministic gradient descent in function values.

show abstract

“…Inexact Hessian information is considered in [16,5,30,31], approximate gradient and Hessian evaluations are used in [12,15,27,32], function, gradient and Hessian values are sampled in [22,6]. The amount of inexactness allowed is controlled dynamically in [12,15,22,16,5].Contributions. The present paper proposes an extension of the unifying framework of [13] for unconstrained or inexpensively-constrained problems that allows inexact evaluations of the objective function and of the required derivatives, in an adaptive way inspired by the trust-region scheme of [17, Section 10.6].…”

mentioning

confidence: 99%

Adaptive Regularization Algorithms with Inexact Evaluations for Nonconvex Optimization

et al. 2019

View full text Add to dashboard Cite

A regularization algorithm using inexact function values and inexact derivatives is proposed and its evaluation complexity analyzed. This algorithm is applicable to unconstrained problems and to problems with inexpensive constraints (that is constraints whose evaluation and enforcement has negligible cost) under the assumption that the derivative of highest degree is β-Hölder continuous. It features a very flexible adaptive mechanism for determining the inexactness which is allowed, at each iteration, when computing objective function values and derivatives. The complexity analysis covers arbitrary optimality order and arbitrary degree of available approximate derivatives. It extends results of Cartis, Gould and Toint [Sharp worst-case evaluation complexity bounds for arbitraryorder nonconvex optimization with inexpensive constraints, arXiv:1811.01220, 2018] on the evaluation complexity to the inexact case: if a q-th order minimizer is sought using approximations to the first p derivatives, it is proved that a suitable approximate minimizer within ǫ is computed by the proposed algorithm in at most O ǫ − p+β p−q+β iterations and at most O | log(ǫ)|ǫ − p+β p−q+β approximate evaluations. An algorithmic variant, although more rigid in practice, can be proved to find such an approximate minimizer in O | log(ǫ)| + ǫ − p+β p−q+β evaluations. While the proposed framework remains so far conceptual for high degrees and orders, it is shown to yield simple and computationally realistic inexact methods when specialized to the unconstrained and bound-constrained first-and second-order cases. The deterministic complexity results are finally extended to the stochastic context, yielding adaptive sample-size rules for subsampling methods typical of machine learning. Bellavia, Gurioli, Morini, Toint: Adaptive Regularization Algorithms with Inexact Evaluations2riving formal bounds on the number of evaluations of the objective function (and possibly of its derivatives) necessary to obtain approximate optimal solutions within a user-specified accuracy. Until recently, the results had focused on methods using first-and second-order derivatives of the objective function, and on convergence guarantees to first-or second-order stationary points [29,23,24,19,11]. Among these contributions, [24,11] analyzed the "regularization method", in which a model of the objective function around a given iterate is constructed by adding a regularization term to the local Taylor expansion, model which is then approximately minimized in an attempt to find a new point with a significantly lower objective function value [21]. Such methods have been shown to possess optimal evaluation complexity [14] for first-and second-order models and minimizers, and have generated considerable interest in the research community. A theoretically significant step was made in [7] for unconstrained problems, where evaluation complexity bounds were obtained for convergence to first-order stationary points of a simplified regularization method using models of arbitrary degree...

show abstract

Global convergence rate analysis of unconstrained optimization methods based on probabilistic models

Cited by 121 publications

References 19 publications

Adaptive Sampling Strategies for Stochastic Optimization

Adaptive Sampling Strategies for Stochastic Optimization

A Stochastic Line Search Method with Expected Complexity Analysis

Adaptive Regularization Algorithms with Inexact Evaluations for Nonconvex Optimization

Contact Info

Product

Resources

About