On the Properties of the Method of Minimization for Convex Functions with Relaxation on the Distance to Extremum

Крутиков, В. Н.; Samoilenko, Natalia S.; Мешечкин, Владимир Викторович

doi:10.1134/s0005117919010090

Cited by 9 publications

(11 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…None of problems 1, 2, 4 and 5 can be solved by the multistep minimization method [24] for n ≥ 100, which emphasizes the relevance of methods with a change in the space metric, in particular, space dilation minimization algorithms capable of solving nonsmooth minimization problems with a high degree of level surface elongation.…”

Section: Computational Experiments Resultsmentioning

confidence: 99%

“…Functions 1, 2, 4, 5 have a significant degree of level surface elongation. The problems of minimizing these functions could not be solved by the multistep minimization methods investigated in [24], which emphasize the relevance of developing methods with a change in the space metric, in particular, space dilation minimization algorithms capable of solving nonsmooth minimization problems with a high degree of level surface elongation.…”

Section: Computational Experiments Resultsmentioning

confidence: 99%

“…For large values of the parameter a in (24), the complexity of solving the minimization problem increases significantly.…”

Section: Formalization Of the Separable Sets Modelmentioning

confidence: 99%

“…Initially, relaxation subgradient minimization methods (RSMMs) were considered in [18][19][20]. They were later developed into a number of effective approaches such as the subgradient method with space dilation in the subgradient direction [21,22] that involve relaxation by distance to the extremum [17,23,24]. The idea of space dilation is to change the metric of the space at each iteration with a linear transformation and to use the direction opposite to that of the subgradient in the space with the transformed metric.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Relaxation Subgradient Algorithms with Machine Learning Procedures

et al. 2022

View full text Add to dashboard Cite

In the modern digital economy, optimal decision support systems, as well as machine learning systems, are becoming an integral part of production processes. Artificial neural network training as well as other engineering problems generate such problems of high dimension that are difficult to solve with traditional gradient or conjugate gradient methods. Relaxation subgradient minimization methods (RSMMs) construct a descent direction that forms an obtuse angle with all subgradients of the current minimum neighborhood, which reduces to the problem of solving systems of inequalities. Having formalized the model and taking into account the specific features of subgradient sets, we reduced the problem of solving a system of inequalities to an approximation problem and obtained an efficient rapidly converging iterative learning algorithm for finding the direction of descent, conceptually similar to the iterative least squares method. The new algorithm is theoretically substantiated, and an estimate of its convergence rate is obtained depending on the parameters of the subgradient set. On this basis, we have developed and substantiated a new RSMM, which has the properties of the conjugate gradient method on quadratic functions. We have developed a practically realizable version of the minimization algorithm that uses a rough one-dimensional search. A computational experiment on complex functions in a space of high dimension confirms the effectiveness of the proposed algorithm. In the problems of training neural network models, where it is required to remove insignificant variables or neurons using methods such as the Tibshirani LASSO, our new algorithm outperforms known methods.

show abstract

Section: Computational Experiments Resultsmentioning

confidence: 99%

Section: Computational Experiments Resultsmentioning

confidence: 99%

“…For large values of the parameter a in (24), the complexity of solving the minimization problem increases significantly.…”

Section: Formalization Of the Separable Sets Modelmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Relaxation Subgradient Algorithms with Machine Learning Procedures

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Methods of this class are applicable to a wide range of problems. A number of effective approaches in the field of non-smooth optimization arose as a result of the creation of the first subgradient methods with space dilation [35,36], in the class of minimization methods relaxing both in function and in distance to the extremum [25,37,38].…”

Section: Introductionmentioning

confidence: 99%

Newtonian Property of Subgradient Method with Optimization of Metric Matrix Parameter Correction

Tovbis,

Krutikov,

Kazakovtsev

2024

Mathematics

View full text Add to dashboard Cite

The work proves that under conditions of instability of the second derivatives of the function in the minimization region, the estimate of the convergence rate of Newton’s method is determined by the parameters of the irreducible part of the conditionality degree of the problem. These parameters represent the degree of difference between eigenvalues of the matrices of the second derivatives in the coordinate system, where this difference is minimal, and the resulting estimate of the convergence rate subsequently acts as a standard. The paper studies the convergence rate of the relaxation subgradient method (RSM) with optimization of the parameters of two-rank correction of metric matrices on smooth strongly convex functions with a Lipschitz gradient without assumptions about the existence of second derivatives of the function. The considered RSM is similar in structure to quasi-Newton minimization methods. Unlike the latter, its metric matrix is not an approximation of the inverse matrix of second derivatives but is adjusted in such a way that it enables one to find the descent direction that takes the method beyond a certain neighborhood of the current minimum as a result of one-dimensional minimization along it. This means that the metric matrix enables one to turn the current gradient into a direction that is gradient-consistent with the set of gradients of some neighborhood of the current minimum. Under broad assumptions on the parameters of transformations of metric matrices, an estimate of the convergence rate of the studied RSM and an estimate of its ability to exclude removable linear background are obtained. The obtained estimates turn out to be qualitatively similar to estimates for Newton’s method. In this case, the assumption of the existence of second derivatives of the function is not required. A computational experiment was carried out in which the quasi-Newton BFGS method and the subgradient method under study were compared on various types of smooth functions. The testing results indicate the effectiveness of the subgradient method in minimizing smooth functions with a high degree of conditionality of the problem and its ability to eliminate the linear background that worsens the convergence.

show abstract