A new two-step gradient-type method for large-scale unconstrained optimization

Abstract

a b s t r a c tIn this paper, we propose some improvements on a new gradient-type method for solving large-scale unconstrained optimization problems, in which we use data from two previous steps to revise the current approximate Hessian. The new method which we considered, resembles to that of Barzilai and Borwein (BB) method. The innovation features of this approach consist in using approximation of the Hessian in diagonal matrix form based on the modified weak secant equation rather than the multiple of the identity matrix in the BB method. Using this approach, we can obtain a higher order accuracy of Hessian approximation when compares to other existing BB-type method. By incorporating a simple monotone strategy, the global convergence of the new method is achieved. Practical insights into the effectiveness of the proposed method are given by numerical comparison with the BB method and its variant.

An improved multi-step gradient-type method for large scale optimization

“…It was shown how ''multi-step'' methods (employing data from previous iteration) may be constructed by means of interpolating polynomials, leading to generalization of the secant equation [7] or weak secant equation [6]. Now our aim is to obtain a higher order accuracy of Hessian approximation in diagonal form, by choosing a different strategy for interpolating the curve in the variable space while keeping the time and space complexity of our proposed method to only O(n) which is the same to that of one-step gradient method.…”

“…(In [6], the weighting matrix is set as B k .) We use a two-step implicit update and employ the following notation to describe the method…”

“…Farid et al [6] incorporated the feature of the two-step multi-point approach into their algorithm, namely TMDGrad and produced an improvement over the algorithm that based upon the commonly used one-step two-point approach. However, depending on the measurement of distances that is required to parameterize the interpolating polynomials via a norm defined by a positive-definite matrix M, the performance of two-step algorithms can be strongly influenced.…”

“…To incorporate a higher order accuracy in approximating the Hessian matrix of objective function, Farid et al [6] proposed to use a two-step multi-point approach (see [7]) in generating the diagonal updating instead of a standard one-step two-point approach. Besides letting the Hessian approximation B k+1 to satisfy the weak secant equation (4), it is also constrained to a relation of the form…”

“…An updating matrix B k+1 that satisfying (5) is then obtained by the following diagonal updating formula (see [6] for details)…”