Derivative-free optimization: a review of algorithms and comparison of software implementations

Abstract

This paper addresses the solution of bound-constrained optimization problems using algorithms that require only the availability of objective function values but no derivative information. We refer to these algorithms as derivative-free algorithms. Fueled by a growing number of applications in science and engineering, the development of derivativefree optimization algorithms has long been studied, and it has found renewed interest in recent time. Along with many derivative-free algorithms, many software implementations have also appeared. The paper presents a review of derivative-free algorithms, followed by a systematic comparison of 22 related implementations using a test set of 502 problems. The test bed includes convex and nonconvex problems, smooth as well as nonsmooth problems. The algorithms were tested under the same conditions and ranked under several criteria, including their ability to find near-global solutions for nonconvex problems, improve a given starting point, and refine a near-optimal solution. A total of 112,448 problem instances were solved. We find that the ability of all these solvers to obtain good solutions diminishes with increasing problem size. For the problems used in this study, TOMLAB/MULTI-MIN, TOMLAB/GLCCLUSTER, MCS and TOMLAB/LGO are better, on average, than other derivative-free solvers in terms of solution quality within 2,500 function evaluations. These global solvers outperform local solvers even for convex problems. Finally, TOMLAB/OQNLP, NEWUOA, and TOMLAB/MULTIMIN show superior performance in terms of refining a nearoptimal solution.

A parallel evolution strategy for an earth imaging problem in geophysics

“…Evolution Strategies are heuristic methods designed for the solution of global optimization problems (with continuous variables) that have performed well in terms of the quality of the final point computed (see [2,3,22,39]). However, like any other method for global optimization, ES's suffer from the curse of dimensionality, meaning that their performance is satisfactory on low dimensional problems, but deteriorates as the dimensionality of the search space increases [29].…”

“…CMA-ES [23] (where CMA stands for Covariance Matrix Adaptation) is regarded as one of the best in the class (µ, λ)-ES in terms of numerical performance [2,3,22,39]. More precisely, CMA-ES belongs to the ES family denoted by (µ/µ W , λ)-ES, where the subscript 'W' indicates the use of 'recombination' via weights.…”

“…CMA-ES has exhibited robust performance for difficult ill-conditioned, non-separable, and highly multi-modal problems [2,3,22,39]. Its main drawback is, however, that a large budget is required to provide outstanding results.…”

Global optimization advances in Mixed-Integer Nonlinear Programming, MINLP, and Constrained Derivative-Free Optimization, CDFO

“…More recently, significant advances in mathematical analysis, computer power, automatic differentiation and global optimization theory and algorithms have en-abled the optimization of complex and large nonlinear problems with theoretical guarantee. However, there is still high interest in CDFO methods because they are suitable for problems which deterministic global optimization methods are unable to handle due to lack of information, noise, non-smoothness and discontinuities (Kolda et al, 2003;Conn et al, 2009b;Rios and Sahinidis, 2013;Martelli and Amaldi, 2014). In the first textbook on Derivative-Free Optimization, Conn et al (2009b) recognize that optimization without derivatives is one of the very challenging open problems in science and engineering, which has a vast number of potential practical applications.…”

On the convergence rate issues of general Markov search for global minimum

“…Furthermore, the global minimal value is usually unknown. The derivative may not exist or may be unavailable (for instance, in case of so called "black box " problems, usually all one have is the possibility of compute the value f (x) at given state x ∈ A and this computation often requires much effort ), and hence many methods belong to the class of derivative-free algorithms, [27]. Because the given method uses poor information on f , its convergence may have very undesired properties based on the following issue: the closer to the optimum, the harder to generate a "better" (in sense of the cost function) state.…”