A Transformation of Accelerated Double Step Size Method for Unconstrained Optimization

Abstract

A reduction of the originally double step size iteration into the single step length scheme is derived under the proposed condition that relates two step lengths in the accelerated double step size gradient descent scheme. The proposed transformation is numerically tested. Obtained results confirm the substantial progress in comparison with the single step size accelerated gradient descent method defined in a classical way regarding all analyzed characteristics: number of iterations, CPU time, and number of function evaluations. Linear convergence of derived method has been proved.

Hybrid Modification of Accelerated Double Direction Method

“…A common way to determine this parameter is through the features of the second-order Taylor's series taken on appropriate scheme (6). Acceleration parameters that were computed in such way are applied in the methods described in [1][2][3][4][5]. According to the iteration form (6), we can conclude that the accelerated gradient methods are of the quasi-Newton type in which the approximation of the Hessian, i.e., its inverse, is obtained by the scalar matrix , where is appropriate identity matrix and = ( , −1 ) is the matching acceleration parameter.…”

Initial Improvement of the Hybrid Accelerated Gradient Descent Process

“…where η > 0 is a constant. The methods presented in [4,6,9,10] can be classified as methods of quasi-Newton type with accelerated approximation of the Hessian inverse, equipped with the line search technique. As in [9], we refer to these methods simply as accelerated gradient descent algorithms with line search.…”

“…This so-called acceleration parameter is calculated from the second-order Taylor series of the relevant iteration at two successive points. So, unlike the model (1.5) which defined the vector direction of conjugate gradient methods, in accelerated gradient descent methods [4,6,9,10] the vector direction is the product of the negative gradient vector and a derived accelerated parameter.…”

“…There are several conjugate and modified gradient descent algorithms [2,7,8,11], as well as accelerated gradient models [4,6,10], comparable to the HSM method and its modifications. We can make some comparisons between the various approaches in terms of the vector direction d k and the step-size value t k .…”

Implementation of gradient methods for optimization of underage costs in aviation industry

“…There are several iterative methods, each defined in a specific way, relevant for this work. Some of them are presented in articles (Andrei, 2006), (Stanimirović et al, 2010), (Petrović et al, 2014), , (Stanimirović et. al.…”