Machine Learning Algorithms of Relaxation Subgradient Method with Space Extension

Крутиков, В. Н.; Мешечкин, Владимир Викторович; Kagan, Elena; Kazakovtsev, Lev

doi:10.1007/978-3-030-77876-7_32

Cited by 3 publications

(10 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, regardless of which of equations ( 14) or ( 19) is used to transform the matrices, the result ( 29) is valid for a sequence of vectors composed of g k or y k , depending on which one of them is used to transform the matrices. Let us show that for Algorithm 1 with fixed values of parameter α k , estimates similar to (29) are valid, and we obtain expressions for the admissible parameters α k in ( 14), (19), at which the values (∆ k , A k ∆ k ) do not increase.…”

Section: Machine Learning Algorithm With Space Dilation For Solving S...mentioning

confidence: 92%

“…In this study, which is an extension of previous work [1], a problem of minimizing a convex, not necessarily differentiable function f (x), x ∈ R n (where R n is a finitedimensional Euclidean space) is discovered. Such a problem is quite common in the field of machine learning (ML), where optimization methods, in particular, gradient descent, are widely used to minimize the loss function during training stage.…”

Section: Introductionmentioning

confidence: 95%

“…In this section, we briefly introduce an algorithm for solving systems of inequalities from [1] and theoretically justify iterations (18) and (19). Specific operators will be used for transformations (13), ( 14) and ( 17)- (19).…”

Section: Machine Learning Algorithm With Space Dilation For Solving S...mentioning

confidence: 99%

“…In order to obtain the visually analyzed operation of the algorithm, similarly to the analysis of iterations of the process ( 9), ( 10) carried out on the basis of Figure 2, we pass to the coordinate system ŝ = A 1/2 k s. In this new coordinate system, corresponding vectors and matrices of iterations ( 13), ( 14) and ( 18), ( 19) are transformed as follows [1]:…”

Section: Machine Learning Algorithm With Space Dilation For Solving S...mentioning

confidence: 99%

“…Embedding the ideas of machine learning theory [25] into such optimization methods made it possible to identify the principles of organizing RSMM with space dilation [26][27][28][29]. The problem of finding the descent direction in the RSMM can be reduced to the problem of solving a system of inequalities on subgradient sets, mathematically formulated as a problem of minimizing a quality functional.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Relaxation Subgradient Algorithms with Machine Learning Procedures

et al. 2022

View full text Add to dashboard Cite

In the modern digital economy, optimal decision support systems, as well as machine learning systems, are becoming an integral part of production processes. Artificial neural network training as well as other engineering problems generate such problems of high dimension that are difficult to solve with traditional gradient or conjugate gradient methods. Relaxation subgradient minimization methods (RSMMs) construct a descent direction that forms an obtuse angle with all subgradients of the current minimum neighborhood, which reduces to the problem of solving systems of inequalities. Having formalized the model and taking into account the specific features of subgradient sets, we reduced the problem of solving a system of inequalities to an approximation problem and obtained an efficient rapidly converging iterative learning algorithm for finding the direction of descent, conceptually similar to the iterative least squares method. The new algorithm is theoretically substantiated, and an estimate of its convergence rate is obtained depending on the parameters of the subgradient set. On this basis, we have developed and substantiated a new RSMM, which has the properties of the conjugate gradient method on quadratic functions. We have developed a practically realizable version of the minimization algorithm that uses a rough one-dimensional search. A computational experiment on complex functions in a space of high dimension confirms the effectiveness of the proposed algorithm. In the problems of training neural network models, where it is required to remove insignificant variables or neurons using methods such as the Tibshirani LASSO, our new algorithm outperforms known methods.

show abstract

Section: Machine Learning Algorithm With Space Dilation For Solving S...mentioning

confidence: 92%

Section: Introductionmentioning

confidence: 95%

Section: Machine Learning Algorithm With Space Dilation For Solving S...mentioning

confidence: 99%

Section: Machine Learning Algorithm With Space Dilation For Solving S...mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Relaxation Subgradient Algorithms with Machine Learning Procedures

et al. 2022

View full text Add to dashboard Cite

show abstract

A Hybrid Differential Evolution for Non-Smooth Optimization Problems

Egorova,

Kazakovtsev,

Krutikov

et al. 2023

FU Math Inform

View full text Add to dashboard Cite

Solving high dimentional, multimodal, non-smooth global optimization problems faces challenges concerning quality of solution, computational costs or even the impossibility of solving the problem. Evolutionary algorithms, in particular, differential evolution algorithm proved itself as good method of global optimization. On the other side, approach based on subgradient methods are good for optimizing non-smooth functions. Combination of these two approaches enables to improve the quality of the algorithm, using the best features of both methods. In this paper, a new hybrid evolutionary approach based on differential evolution and subgradient algorithm as the local search procedure is proposed. Behavior of the proposed SSGDE algorithm was studied in a numerical experiment on three groups of generated tests. Comparison of the new hybrid algorithm with the pure DE approach showed the advantage of the SSGDE. It has been experimentally established that the proposed method finds the global minimum in the best way for all considered dimensions of the problem with respect to the differential evolution method. The SSGDE algorithm showed the best results with a significant increase in the number of functions.

show abstract

Properties of the Quadratic Transformation of Dual Variables

Крутиков

Tovbis²,

Быков

et al. 2023

Algorithms

View full text Add to dashboard Cite

We investigate a solution of a convex programming problem with a strongly convex objective function based on the dual approach. A dual optimization problem has constraints on the positivity of variables. We study the methods and properties of transformations of dual variables that enable us to obtain an unconstrained optimization problem. We investigate the previously known method of transforming the components of dual variables in the form of their modulus (modulus method). We show that in the case of using the modulus method, the degree of the degeneracy of the function increases as it approaches the optimal point. Taking into account the ambiguity of the gradient in the boundary regions of the sign change of the new dual function variables and the increase in the degree of the function degeneracy, we need to use relaxation subgradient methods (RSM) that are difficult to implement and that can solve non-smooth non-convex optimization problems with a high degree of elongation of level surfaces. We propose to use the transformation of the components of dual variables in the form of their square (quadratic method). We prove that the transformed dual function has a Lipschitz gradient with a quadratic method of transformation. This enables us to use efficient gradient methods to find the extremum. The above properties are confirmed by a computational experiment. With a quadratic transformation compared to a modulus transformation, it is possible to obtain a solution of the problem by relaxation subgradient methods and smooth function minimization methods (conjugate gradient method and quasi-Newtonian method) with higher accuracy and lower computational costs. The noted transformations of dual variables were used in the program module for calculating the maximum permissible emissions of enterprises (MPE) of the software package for environmental monitoring of atmospheric air (ERA-AIR).

show abstract

Machine Learning Algorithms of Relaxation Subgradient Method with Space Extension

Cited by 3 publications

References 16 publications

Relaxation Subgradient Algorithms with Machine Learning Procedures

Relaxation Subgradient Algorithms with Machine Learning Procedures

A Hybrid Differential Evolution for Non-Smooth Optimization Problems

Properties of the Quadratic Transformation of Dual Variables

Contact Info

Product

Resources

About