On the Numerical Performance of Derivative-Free Optimization Methods Based on Finite-Difference Approximations

Shi, Hao-Jun Michael; Xuan, Melody Qiming; Oztoprak, Figen; Nocedal, Jorge

doi:10.48550/arxiv.2102.09762

Cited by 5 publications

(5 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The study of nonlinear optimization problems with errors or noise in the function and gradient has attracted attention in recent years, motivated by the use of finite difference approximations to derivatives [19,26,25] and by applications in machine learning; see [15] for a review of some recent work.…”

Section: Literature Reviewmentioning

confidence: 99%

A Trust Region Method for the Optimization of Noisy Functions

Sun¹,

Nocedal²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Classical trust region methods were designed to solve problems in which function and gradient information are exact. This paper considers the case when there are bounded errors (or noise) in the above computations and proposes a simple modification of the trust region method to cope with these errors. The new algorithm only requires information about the size of the errors in the function evaluations and incurs no additional computational expense. It is shown that, when applied to a smooth (but not necessarily convex) objective function, the iterates of the algorithm visit a neighborhood of stationarity infinitely often, and that the rest of the sequence cannot stray too far away, as measured by function values. Numerical results illustrate how the classical trust region algorithm may fail in the presence of noise, and how the proposed algorithm ensures steady progress towards stationarity in these cases.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

A Trust Region Method for the Optimization of Noisy Functions

Sun¹,

Nocedal²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…That said, one of the strengths of direct-search methods is that they are more readily applicable when an objective function is nonsmooth [2,14] or even discontinuous [31]. Finite-difference approaches approximate derivatives using finite difference schemes, which are then embedded within a gradient-based optimization approach, such as a steepest descent or quasi-Newton method; see, e.g., [30]. Empirical evidence has shown that finite-difference methods can be competitive with model-based methods, at least when one presumes no noise in the function values.…”

Section: ≥0mentioning

confidence: 99%

Derivative-Free Bound-Constrained Optimization for Solving Structured Problems with Surrogate Models

Curtis¹,

Dezfulian²,

Wächter³

2022

Preprint

View full text Add to dashboard Cite

A derivative-free optimization (DFO) algorithm is presented. The distinguishing feature of the algorithm is that it allows for the use of function values that have been made available through prior runs of a DFO algorithm for solving prior related optimization problems. Applications in which sequences of related optimization problems are solved such that the proposed algorithm is applicable include certain least-squares and simulation-based optimization problems. A convergence guarantee of a generic algorithmic framework that exploits prior function evaluations is presented, then a particular instance of the framework is proposed and analyzed. The results of numerical experiments when solving engineering test problems show that the algorithm gains advantage over a state-of-the-art DFO algorithm when prior function values are available.

show abstract

“…Several suggested methods have given fair outcomes for computing the gradient vector values numerically. See [63][64][65][66][67].…”

Section: Numerical Differentiationmentioning

confidence: 99%

A Family of Hybrid Stochastic Conjugate Gradient Algorithms for Local and Global Minimization Problems

et al. 2022

View full text Add to dashboard Cite

This paper contains two main parts, Part I and Part II, which discuss the local and global minimization problems, respectively. In Part I, a fresh conjugate gradient (CG) technique is suggested and then combined with a line-search technique to obtain a globally convergent algorithm. The finite difference approximations approach is used to compute the approximate values of the first derivative of the function f. The convergence analysis of the suggested method is established. The comparisons between the performance of the new CG method and the performance of four other CG methods demonstrate that the proposed CG method is promising and competitive for finding a local optimum point. In Part II, three formulas are designed by which a group of solutions are generated. This set of random formulas is hybridized with the globally convergent CG algorithm to obtain a hybrid stochastic conjugate gradient algorithm denoted by HSSZH. The HSSZH algorithm finds the approximate value of the global solution of a global optimization problem. Five combined stochastic conjugate gradient algorithms are constructed. The performance profiles are used to assess and compare the rendition of the family of hybrid stochastic conjugate gradient algorithms. The comparison results between our proposed HSSZH algorithm and four other hybrid stochastic conjugate gradient techniques demonstrate that the suggested HSSZH method is competitive with, and in all cases superior to, the four algorithms in terms of the efficiency, reliability and effectiveness to find the approximate solution of the global optimization problem that contains a non-convex function.

show abstract

On the Numerical Performance of Derivative-Free Optimization Methods Based on Finite-Difference Approximations

Cited by 5 publications

References 18 publications

A Trust Region Method for the Optimization of Noisy Functions

A Trust Region Method for the Optimization of Noisy Functions

Derivative-Free Bound-Constrained Optimization for Solving Structured Problems with Surrogate Models

A Family of Hybrid Stochastic Conjugate Gradient Algorithms for Local and Global Minimization Problems

Contact Info

Product

Resources

About