Piece-wise quadratic approximations of arbitrary error functions for fast and robust machine learning

Abstract

Most of machine learning approaches have stemmed from the application of minimizing the mean squared distance principle, based on the computationally efficient quadratic optimization methods. However, when faced with high-dimensional and noisy data, the quadratic error functionals demonstrated many weaknesses including high sensitivity to contaminating factors and dimensionality curse. Therefore, a lot of recent applications in machine learning exploited properties of non-quadratic error functionals based on L 1 norm or even sub-linear potentials corresponding to quasinorms L p (0 < p < 1). The back side of these approaches is increase in computational cost for optimization. Till so far, no approaches have been suggested to deal with arbitrary error functionals, in a flexible and computationally efficient framework. In this paper, we develop a theory and basic universal data approximation algorithms (k-means, principal components, principal manifolds and graphs, regularized and sparse regression), based on piece-wise quadratic error potentials of subquadratic growth (PQSQ potentials). We develop a new and universal framework to minimize arbitrary sub-quadratic error potentials using an algorithm with guaranteed fast convergence to the local or global error minimum. The theory of PQSQ potentials is based on the notion of the cone of minorant functions, and represents a natural approximation formalism based on the application of min-plus algebra. The approach can be applied in most of existing machine learning methods, including methods of data approximation and regularized and sparse regression, leading to the improvement in the computational cost/accuracy trade-off. We demonstrate that on synthetic and real-life datasets PQSQ-based machine learning methods achieve orders of magnitude faster computational performance than the corresponding state-of-the-art methods, having similar or better approximation accuracy.

Robust and Scalable Learning of Complex Intrinsic Dataset Geometry via ElPiGraph

“…The approach is similar to the ‘data-driven’ trimmed k -means clustering [ 50 ]. Alternative ways of constructing robust principal graphs include using piece-wise quadratic subquadratic error functions (PQSQ potentials) [ 51 ], which uses computationally efficient non-quadratic error functions for approximating a dataset. These two approaches will be implemented in the future versions of ElPiGraph.…”

Search for Errors in the Base of Navigation Data by the Method of Spline Isosurface Visualization

“…Авторская идея заключается в представлении базы цифровых данных в виде визуализированной компьютерной модели с реализацией возможности выявления неформализованного признака между навигационными параметрами и синтезированного на их матричной основе объемного геометрического образа [21]. Данный подход позволяет получить компромиссное решение при невозможности формулирования формальных критериев нечеткого поиска ошибочности [22], [23].…”

Blessing of dimensionality: mathematical foundations of the statistical physics of data

“…[25,26]. Even non-convex quasinorms and their tropical approximations are used efficiently to provide sparse and robust learning results [32]. Vapnik [26] defined a formalized fragment of machine learning using minimization of a risk functional that is the mathematical expectation of a general loss function.…”