A survey of gradient methods for solving nonlinear optimization

Stanimirović, Predrag S.; Mathematics, Višegradska; Ivanov, Branislav; Ma, Haifeng; Mosić, Dijana

doi:10.3934/era.2020115

Cited by 22 publications

(11 citation statements)

References 125 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…k ≥ 1. More details about accelerated gradient methods can be found in [19,22,23]. Since ℷ k ′ (α k ) � 0 for α k � (1/2) and ℷ k (0) � ℷ k (1) � 1, mathematical analysis of the function ℷ k (α k ) in the interval α k ∈ (0, 1] reveals maxℷ k (α k ) � ℷ k (1/2) � (5/4) and 1 ≤ ℷ k ≤ (5/4).…”

Section: Introduction and Overview Of Related Resultsmentioning

confidence: 99%

A Modified Dai–Liao Conjugate Gradient Method Based on a Scalar Matrix Approximation of Hessian and Its Application

Ivanov

Milovanović

Stanimirović

et al. 2023

Journal of Mathematics

Self Cite

View full text Add to dashboard Cite

We introduce and investigate proper accelerations of the Dai–Liao (DL) conjugate gradient (CG) family of iterations for solving large-scale unconstrained optimization problems. The improvements are based on appropriate modifications of the CG update parameter in DL conjugate gradient methods. The leading idea is to combine search directions in accelerated gradient descent methods, defined based on the Hessian approximation by an appropriate diagonal matrix in quasi-Newton methods, with search directions in DL-type CG methods. The global convergence of the modified Dai–Liao conjugate gradient method has been proved on the set of uniformly convex functions. The efficiency and robustness of the newly presented methods are confirmed in comparison with similar methods, analyzing numerical results concerning the CPU time, a number of function evaluations, and the number of iterative steps. The proposed method is successfully applied to deal with an optimization problem arising in 2D robotic motion control.

show abstract

Section: Introduction and Overview Of Related Resultsmentioning

confidence: 99%

A Modified Dai–Liao Conjugate Gradient Method Based on a Scalar Matrix Approximation of Hessian and Its Application

Ivanov

Milovanović

Stanimirović

et al. 2023

Journal of Mathematics

Self Cite

View full text Add to dashboard Cite

show abstract

“…This is a similar equation as in the gradient descent method with the exact line search [31]. By using (18), we can always find α that assures modulus reduction.…”

Section: Algorithm 2: Computation Of N Pmentioning

confidence: 99%

“…This method may give the best value of α, but it is computationally too expensive in practice. Instead, we can use a method similar to the backtracking line search known in optimisation [31]. The proposed method is presented in Algorithm 3.…”

Section: Algorithm 2: Computation Of N Pmentioning

confidence: 99%

On the robust Newton’s method with the Mann iteration and the artistic patterns from its dynamics

2021

View full text Add to dashboard Cite

There are two main aims of this paper. The first one is to show some improvement of the robust Newton’s method (RNM) introduced recently by Kalantari. The RNM is a generalisation of the well-known Newton’s root finding method. Since the base method is undefined at critical points, the RNM allows working also at such points. In this paper, we improve the RNM method by applying the Mann iteration instead of the standard Picard iteration. This leads to an essential decrease in the number of root finding steps without visible destroying the sharp boundaries among the basins of attractions presented in polynomiographs. Furthermore, we investigate visually the dynamics of the RNM with the Mann iteration together with the basins of attraction for varying Mann’s iteration parameter with the help of polynomiographs for several polynomials. The second aim of this paper is to present the intriguing polynomiographs obtained from the dynamics of the RNM with the Mann iteration under various sequences used in this iteration. The obtained polynomiographs differ considerably from the ones obtained with the RNM and are interesting from the artistic perspective. Moreover, they can easily find applications in wallpaper or fabric design.

show abstract

“…In a wide range of practical disciplines such as machine learning and signal processing, large scale continuous optimization models emerge, often as the unconstrained optimization problem min 𝑥∈R 𝑛 𝑓 (𝑥), (1.1) where the objective function 𝑓 is here assumed to be smooth. Scholar studies reflect the value of the CG techniques among the various continuous optimization algorithms [95]. Initially founded by Hestenes and Stiefel (HS) [59] in the midst of the previous century for solving positive definite systems of linear equations, and then adopted by Fletcher and Reeves [51] for the unconstrained optimization, CG algorithms benefit the low memory storage and simple iterations squarely as well as the second order information implicitly [57,83].…”

Section: Introductionmentioning

confidence: 99%

A survey on the Dai–Liao family of nonlinear conjugate gradient methods

Babaie-Kafaki

2023

RAIRO-Oper. Res.

View full text Add to dashboard Cite

At the beginning of this century, which is characterized by huge flows of emerging data, Dai and Liao proposed a pervasive conjugacy condition that triggered the interest of many optimization scholars. Recognized as a sophisticated conjugate gradient (CG) algorithm after about two decades, here we share our visions and thoughts on the method in the framework of a review study. In this regard, we first discuss the modified Dai–Liao methods based on the modified secant equations given in the literature, mostly with the aim of applying the objective function values in addition to the gradient information. Then, several adaptive, sort of optimal choices for the parameter of the method are studied. Especially, we devote a part of our study to the modified versions of the Hager–Zhang and Dai–Kou CG algorithms, being well-known members of the Dai–Liao class of CG methods. Extensions of the classical CG methods based on the Dai–Liao approach are also reviewed. Finally, we discuss the optimization models of practical disciplines that have been addressed by the Dai–Liao approach, including the nonlinear systems of equations, image restoration and compressed sensing.

show abstract

A survey of gradient methods for solving nonlinear optimization

Cited by 22 publications

References 125 publications

A Modified Dai–Liao Conjugate Gradient Method Based on a Scalar Matrix Approximation of Hessian and Its Application

A Modified Dai–Liao Conjugate Gradient Method Based on a Scalar Matrix Approximation of Hessian and Its Application

On the robust Newton’s method with the Mann iteration and the artistic patterns from its dynamics

A survey on the Dai–Liao family of nonlinear conjugate gradient methods

Contact Info

Product

Resources

About