Relaxation Subgradient Algorithms with Machine Learning Procedures

Abstract

In the modern digital economy, optimal decision support systems, as well as machine learning systems, are becoming an integral part of production processes. Artificial neural network training as well as other engineering problems generate such problems of high dimension that are difficult to solve with traditional gradient or conjugate gradient methods. Relaxation subgradient minimization methods (RSMMs) construct a descent direction that forms an obtuse angle with all subgradients of the current minimum neighborhood, which reduces to the problem of solving systems of inequalities. Having formalized the model and taking into account the specific features of subgradient sets, we reduced the problem of solving a system of inequalities to an approximation problem and obtained an efficient rapidly converging iterative learning algorithm for finding the direction of descent, conceptually similar to the iterative least squares method. The new algorithm is theoretically substantiated, and an estimate of its convergence rate is obtained depending on the parameters of the subgradient set. On this basis, we have developed and substantiated a new RSMM, which has the properties of the conjugate gradient method on quadratic functions. We have developed a practically realizable version of the minimization algorithm that uses a rough one-dimensional search. A computational experiment on complex functions in a space of high dimension confirms the effectiveness of the proposed algorithm. In the problems of training neural network models, where it is required to remove insignificant variables or neurons using methods such as the Tibshirani LASSO, our new algorithm outperforms known methods.

Applied and Computational Mathematics for Digital Environments

“…A total of 12 papers were submitted to this Special Issue, of which 11 were published (91.67%) [11][12][13][14][15][16][17][18][19][20][21] and only 1 was rejected (8.33%), indicating the very high quality of the original submissions.…”

“…The study by Krutikov et al [18] is dedicated to a new relaxation subgradient minimization method (RSMM). The computational experiments conducted by the authors confirmed the effectiveness of the proposed algorithm, showing that it outperforms currently known methods.…”

Newtonian Property of Subgradient Method with Optimization of Metric Matrix Parameter Correction

“…This research is a continuation of previous studies [27,28] and aimed at studying the capabilities of the Newton's method and the relaxation subgradient method with optimization of the parameters of rank-two correction of metric matrices [27] to eliminate the linear background that worsens the convergence in the conditions of the existence of transformation V with the properties noted above. Similar studies for quasi-Newton methods were carried out in [29].…”

“…The quantity M G determines the complexity of solving system (26). The transformation parameters of the metric matrices of the subgradient method are found according to expression (28).…”

A Family of Multi-Step Subgradient Minimization Methods

“…The principle of organization in a number of the RSMs [70] is that, in a particular RSM, there is an independent algorithm for finding the descent direction, which makes it possible to go beyond some neighborhood of the current minimum. In [70,71], the problem of finding the descent direction in RSM was formulated as the problem of solving systems of inequalities on separable sets.…”