Efficient Implementation of the Nelder–Mead Search Algorithm

Abstract

The Nelder-Mead or simplex search algorithm is one of the best known algorithms for unconstrained optimization of non-smooth functions. Even though the basic algorithm is quite simple, it is implemented in many different ways. Apart from some minor computational details, the main difference between various implementations lies in the selection of convergence (or termination) tests, which are used to break the iteration process.A fairly simple efficiency analysis of each iteration step reveals a potential computational bottleneck in the domain convergence test. To be efficient, such a test has to be sublinear in the number of vertices of the working simplex. We have tested some of the most common implementations of the Nelder-Mead algorithm, and none of them is efficient in this sense.Therefore, we propose a simple and efficient domain convergence test and discuss some of its properties. This test is based on tracking the volume of the working simplex throughout the iterations. Similar termination tests can also be applied in some other simplex-based direct search methods.

Difference of Gaussians revolved along elliptical paths for ultrasound fetal head segmentation

“…The shrinkage requires the computation of n new points, whereas the expansion, reflection, and contraction require the computation of just one new point. Thus, the latter cases, which are the most common in practice, can be accelerated by sorting the points of the simplex using an insertion sort algorithm because the old points of the simplex are already ordered [23]. Additionally, the new mean point can be computed by updating the old one by subtracting the removed point and adding the new one, instead of averaging all n points [23].…”

Accurate Landmarking of Three-Dimensional Facial Data in the Presence of Facial Expressions and Occlusions Using a Three-Dimensional Statistical Facial Feature Model

“…This method does not require any derivative information and is widely used to solve parameter estimation and statistical problems of similar nature [41].…”

Semi Analytical Formulation of Dispersion Characteristics for a Step and ParabolicIndex Optical Fiber: Introduction of Nelder-Mead Simplex Method for Nonlinear Unconstrained Minimization

“…Various forms of term_x and term_f tests have been used in practice, and some common implementations of the algorithm have only one of these two tests [14,15].…”

“…it is effective and computationally compact. Although the method is relatively old and considering recent advances in direct search methods, the Nelder-Mead method is still among the most popular direct search methods in practice [14,15].…”