Blessing of dimensionality: mathematical foundations of the statistical physics of data

Abstract

The concentrations of measure phenomena were discovered as the mathematical background to statistical mechanics at the end of the nineteenth/beginning of the twentieth century and have been explored in mathematics ever since. At the beginning of the twenty-first century, it became clear that the proper utilization of these phenomena in machine learning might transform the into the This paper summarizes recently discovered phenomena of measure concentration which drastically simplify some machine learning problems in high dimension, and allow us to correct legacy artificial intelligence systems. The classical concentration of measure theorems state that i.i.d. random points are concentrated in a thin layer near a surface (a sphere or equators of a sphere, an average or median-level set of energy or another Lipschitz function, etc.). The new describe the thin structure of these thin layers: the random points are not only concentrated in a thin layer but are all linearly separable from the rest of the set, even for exponentially large random sets. The linear functionals for separation of points can be selected in the form of the linear Fisher's discriminant. All artificial intelligence systems make errors. Non-destructive correction requires separation of the situations (samples) with errors from the samples corresponding to correct behaviour by a simple and robust classifier. The stochastic separation theorems provide us with such classifiers and determine a non-iterative (one-shot) procedure for their construction.This article is part of the theme issue 'Hilbert's sixth problem'.

High-Dimensional Brain in a High-Dimensional World: Blessing of Dimensionality

“…There is a fundamental difference and complementarity between analysis of essentially high-dimensional datasets, where simple linear methods are applicable, and reducible datasets for which non-linear methods are needed, both for reduction and analysis [30]. This alternative in neuroscience was described as high-dimensional 'brainland' versus low-dimensional 'flatland' [70].…”

“…Such a separability is important for the solution of a technological problem of fast, robust and non-damaging correction of AI mistakes [30,39,40]. AI systems make mistakes and will make mistakes in the future.…”

“…The high risk/normal situation classifier is prepared by supervised training on situations with diagnosed errors (universal construction). The online training algorithm could be very simple like Fisher's linear discriminants or their ensembles [30,39,40,43,44]. Correction rules for high-risk situations are specific to a particular problem.…”

“…We just have to use it properly. Each application requires a specific balance between the extraction of important low-dimensional structures ('reduction') and the use of the remarkable properties of high-dimensional geometry that underlie statistical physics and other fundamental results [30,31].Both the curse and the blessing of dimensionality are the consequences of the measure concentration phenomena [30][31][32][33]. These phenomena were discovered in the development of the statistical backgrounds of thermodynamics.…”

Bringing the Blessing of Dimensionality to the Edge

“…In this paper we show that stochastic separation theorems, or the blessing of dimensionality [24], [25], stemming from the concentration of measure effects [26], [27], [28], can be adapted and applied to address these questions. We present and justify both mathematically and experimentally an algorithm capable of delivering the removal of errors at computational costs compatible with deployment at the edge.…”