On the Halley class of methods for unconstrainedoptimization problems

Abstract

Third-order methods can be used to solve efficiently the unconstrained optimization problems, and they, in most cases, use fewer iterations but more computational cost per iteration than a second-order method to reach the same accuracy. Recently, it has been shown by an article that under some conditions the ratio of the number of arithmetic operations of a third-order method (the Halley class of methods) and Newton's method is constant (at most 5) per iteration. Automatic differentiation (AD) can compute fast and accurate derivatives such as the Jacobian, Hessian matrix and the tensor of the function. The Halley class of methods includes these high-order derivatives. In this paper, we apply AD efficiently to the methods and investigate the computational complexity of them. The results show that under general conditions even including the computation of the function and its derivative terms, the upper bound of the ratio can be reduced to 3.5.

New Variants of Newton’s Method for Nonlinear Unconstrained Optimization Problems

“…Its role in optimization theory can not be overestimated as the method is the basis for the most effective procedures in linear and nonlinear programming. For a more detailed survey, one can refer [1][2][3][4] and the references cited therein. The idea behind the Newton's method is to approximate the objective function locally by a quadratic function which agrees with the function at a point.…”

Original Halley Method and its Improvement with Automatic Differentiation

“…However, they may use more computational complexity per iteration than Newton's method. Recently, by efficiently utilizing the structure and using automatic differentiation, Gundersen and Steihaug ([3], [4]) and Zhang [8] show that the ratio of the number of arithmetic operations of Halley class of methods and Newton's method is small constant per iteration, therefore, Halley class of methods are very efficient to solve the optimization problems.…”