Adaptive Finite-Difference Interval Estimation for Noisy Derivative-Free Optimization

Shi, Hao-Jun Michael; Xie, Yuchen; Xuan, Melody Qiming; Nocedal, Jorge

doi:10.1137/21m1452470

Cited by 10 publications

(3 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, given the fact that CG and BFGS take the finite difference parameter h ≈ 10 −8 , their unfavorable performance is not surprising. It does not contradict the observations in [67,68],…”

Section: Stability Under Noisesupporting

confidence: 63%

“…This is because Powell's methods (NEWUOA in this experiment) gradually adjust the geometry of the interpolation set during the iterations, making progress until the interpolation points are too close to distinguish noise from true objective function values. This is not specific to Powell's methods but also applies to other algorithms that sample the objective function on a set of points with adaptively controlled geometry, including finite-difference methods with well-chosen difference parameters [67].…”

Section: Stability Under Noisementioning

confidence: 99%

See 1 more Smart Citation

PDFO: A Cross-Platform Package for Powell's Derivative-Free Optimization Solvers

Ragonneau¹,

Zhang²

2023

Preprint

View full text Add to dashboard Cite

Late Professor M. J. D. Powell designed five trust-region derivative-free optimization methods, namely COBYLA, UOBYQA, NEWUOA, BOBYQA, and LINCOA. He also carefully implemented them into publicly available solvers, which are renowned for their robustness and efficiency. However, the solvers were implemented in Fortran 77 and hence may not be easily accessible to some users. We introduce the PDFO package, which provides user-friendly Python and MATLAB interfaces to Powell's code. With PDFO, users of such languages can call Powell's Fortran solvers easily without dealing with the Fortran code.Moreover, PDFO includes bug fixes and improvements, which are particularly important for handling problems that suffer from ill-conditioning or failures of function evaluations. In addition to the PDFO package, we provide an overview of Powell's methods, sketching them from a uniform perspective, summarizing their main features, and highlighting the similarities and interconnections among them. We also present experiments on PDFO to demonstrate its stability under noise, tolerance of failures in function evaluations, and potential in solving certain hyperparameter optimization problems.

show abstract

Section: Stability Under Noisesupporting

confidence: 63%

Section: Stability Under Noisementioning

confidence: 99%

PDFO: A Cross-Platform Package for Powell's Derivative-Free Optimization Solvers

Ragonneau¹,

Zhang²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In fact, HJ-Prox does not require differentiability of f . Related methods include random gradients ( 21 – 24 ), sparsity-based methods ( 25 – 27 ), derivative-free quasi-Newton methods ( 28 – 30 ), finite-difference-based methods ( 31 , 32 ), numerical quadrature-based methods ( 33 , 34 ), Bayesian methods ( 29 ), and comparison methods ( 35 ). As proximals closely relate to the gradient of Moreau envelopes, our work relates to methods that minimize Moreau envelopes (or their approximations) ( 16 , 19 , 36 – 40 ).…”

Section: Related Workmentioning

confidence: 99%

A Hamilton–Jacobi-based proximal operator

Osher

Heaton²,

Fung

2023

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

First-order optimization algorithms are widely used today. Two standard building blocks in these algorithms are proximal operators (proximals) and gradients. Although gradients can be computed for a wide array of functions, explicit proximal formulas are known for only limited classes of functions. We provide an algorithm, HJ-Prox, for accurately approximating such proximals. This is derived from a collection of relations between proximals, Moreau envelopes, Hamilton–Jacobi (HJ) equations, heat equations, and Monte Carlo sampling. In particular, HJ-Prox smoothly approximates the Moreau envelope and its gradient. The smoothness can be adjusted to act as a denoiser. Our approach applies even when functions are accessible only by (possibly noisy) black box samples. We show that HJ-Prox is effective numerically via several examples.

show abstract

A trust region method for noisy unconstrained optimization

Sun

Nocedal

2023

Math. Program.

View full text Add to dashboard Cite

Adaptive Finite-Difference Interval Estimation for Noisy Derivative-Free Optimization

Cited by 10 publications

References 18 publications

PDFO: A Cross-Platform Package for Powell's Derivative-Free Optimization Solvers

PDFO: A Cross-Platform Package for Powell's Derivative-Free Optimization Solvers

A Hamilton–Jacobi-based proximal operator

A trust region method for noisy unconstrained optimization

Contact Info

Product

Resources

About