Optimization of Subgradient Method Parameters Based on Rank-Two Correction of Metric Matrices

Крутиков, В. Н.; Stanimirović, Predrag S.; Indenko, O. N.; Tovbis, Elena; Kazakovtsev, Lev

doi:10.1134/s1990478922030073

Cited by 5 publications

(31 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This research is a continuation of previous studies [27,28] and aimed at studying the capabilities of the Newton's method and the relaxation subgradient method with optimization of the parameters of rank-two correction of metric matrices [27] to eliminate the linear background that worsens the convergence in the conditions of the existence of transformation V with the properties noted above. Similar studies for quasi-Newton methods were carried out in [29].…”

Section: Introductionmentioning

confidence: 95%

“…In what follows, this estimate serves as a standard, and the ability of a certain method, like Newton's method, to exclude linear background will be called its Newtonian property. The main goal of the work is to substantiate the presence of the Newtonian property in RSM with a change in the space metric [27]. As shown in [29], the noted Newtonian property is inherent in quasi-Newton methods.…”

Section: Introductionmentioning

confidence: 99%

“…In [36], an effective RSM with space dilation in the direction of the subgradient difference (RSMSD) was developed. Subsequent work on the creation of effective RSMs is associated with identifying the origin of RSMSD and its theoretical justification [42,43]. Formalization of the model of subgradient sets and the use of ideas and machine learning algorithms [44] made it possible to identify the principles of organizing RSM with space dilation [43] and obtain a theoretical basis for their creation.…”

Section: Introductionmentioning

confidence: 99%

“…Subsequent work on the creation of effective RSMs is associated with identifying the origin of RSMSD and its theoretical justification [42,43]. Formalization of the model of subgradient sets and the use of ideas and machine learning algorithms [44] made it possible to identify the principles of organizing RSM with space dilation [43] and obtain a theoretical basis for their creation. It turned out that the problem of finding the descent direction in RSM can be reduced to the problem of solving a system of inequalities on subgradient sets and mathematically formulated as a solution to the problem of minimizing a quality functional.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Newtonian Property of Subgradient Method with Optimization of Metric Matrix Parameter Correction

Tovbis,

Krutikov,

Kazakovtsev

2024

Mathematics

View full text Add to dashboard Cite

The work proves that under conditions of instability of the second derivatives of the function in the minimization region, the estimate of the convergence rate of Newton’s method is determined by the parameters of the irreducible part of the conditionality degree of the problem. These parameters represent the degree of difference between eigenvalues of the matrices of the second derivatives in the coordinate system, where this difference is minimal, and the resulting estimate of the convergence rate subsequently acts as a standard. The paper studies the convergence rate of the relaxation subgradient method (RSM) with optimization of the parameters of two-rank correction of metric matrices on smooth strongly convex functions with a Lipschitz gradient without assumptions about the existence of second derivatives of the function. The considered RSM is similar in structure to quasi-Newton minimization methods. Unlike the latter, its metric matrix is not an approximation of the inverse matrix of second derivatives but is adjusted in such a way that it enables one to find the descent direction that takes the method beyond a certain neighborhood of the current minimum as a result of one-dimensional minimization along it. This means that the metric matrix enables one to turn the current gradient into a direction that is gradient-consistent with the set of gradients of some neighborhood of the current minimum. Under broad assumptions on the parameters of transformations of metric matrices, an estimate of the convergence rate of the studied RSM and an estimate of its ability to exclude removable linear background are obtained. The obtained estimates turn out to be qualitatively similar to estimates for Newton’s method. In this case, the assumption of the existence of second derivatives of the function is not required. A computational experiment was carried out in which the quasi-Newton BFGS method and the subgradient method under study were compared on various types of smooth functions. The testing results indicate the effectiveness of the subgradient method in minimizing smooth functions with a high degree of conditionality of the problem and its ability to eliminate the linear background that worsens the convergence.

show abstract

Section: Introductionmentioning

confidence: 95%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Newtonian Property of Subgradient Method with Optimization of Metric Matrix Parameter Correction

Tovbis,

Krutikov,

Kazakovtsev

2024

Mathematics

View full text Add to dashboard Cite

show abstract

“…Embedding the ideas of machine learning theory [25] into such optimization methods made it possible to identify the principles of organizing RSMM with space dilation [26][27][28][29]. The problem of finding the descent direction in the RSMM can be reduced to the problem of solving a system of inequalities on subgradient sets, mathematically formulated as a problem of minimizing a quality functional.…”

Section: Introductionmentioning

confidence: 99%

Relaxation Subgradient Algorithms with Machine Learning Procedures

et al. 2022

View full text Add to dashboard Cite

In the modern digital economy, optimal decision support systems, as well as machine learning systems, are becoming an integral part of production processes. Artificial neural network training as well as other engineering problems generate such problems of high dimension that are difficult to solve with traditional gradient or conjugate gradient methods. Relaxation subgradient minimization methods (RSMMs) construct a descent direction that forms an obtuse angle with all subgradients of the current minimum neighborhood, which reduces to the problem of solving systems of inequalities. Having formalized the model and taking into account the specific features of subgradient sets, we reduced the problem of solving a system of inequalities to an approximation problem and obtained an efficient rapidly converging iterative learning algorithm for finding the direction of descent, conceptually similar to the iterative least squares method. The new algorithm is theoretically substantiated, and an estimate of its convergence rate is obtained depending on the parameters of the subgradient set. On this basis, we have developed and substantiated a new RSMM, which has the properties of the conjugate gradient method on quadratic functions. We have developed a practically realizable version of the minimization algorithm that uses a rough one-dimensional search. A computational experiment on complex functions in a space of high dimension confirms the effectiveness of the proposed algorithm. In the problems of training neural network models, where it is required to remove insignificant variables or neurons using methods such as the Tibshirani LASSO, our new algorithm outperforms known methods.

show abstract