High-Dimensional Brain in a High-Dimensional World: Blessing of Dimensionality

Abstract

High-dimensional data and high-dimensional representations of reality are inherent features of modern Artificial Intelligence systems and applications of machine learning. The well-known phenomenon of the "curse of dimensionality" states: many problems become exponentially difficult in high dimensions. Recently, the other side of the coin, the "blessing of dimensionality", has attracted much attention. It turns out that generic high-dimensional datasets exhibit fairly simple geometric properties. Thus, there is a fundamental tradeoff between complexity and simplicity in high dimensional spaces. Here we present a brief explanatory review of recent ideas, results and hypotheses about the blessing of dimensionality and related simplifying effects relevant to machine learning and neuroscience. of 18programming was considered a method of dimensionality reduction in the optimization of a multi-stage decision process. Bellman returned to the problem of dimensionality reduction many times in different contexts [3]. Now, dimensionality reduction is an essential element of the engineering (the "practical man") approach to mathematical modeling [4]. Many model reduction methods were developed and successfully implemented in applications, from various versions of principal component analysis to approximation by manifolds, graphs, and complexes [5][6][7], and low-rank tensor network decompositions [8,9].Various reasons and forms of the curse of dimensionality were classified and studied, from the obvious combinatorial explosion (for example, for n binary Boolean attributes, to check all the combinations of values we have to analyze 2 n cases) to more sophisticated distance concentration: in a high-dimensional space, the distances between randomly selected points tend to concentrate near their mean value, and the neighbor-based methods of data analysis become useless in their standard forms [10,11]. Many "good" polynomial time algorithms become useless in high dimensions.Surprisingly, however, and despite the expected challenges and difficulties, common-sense heuristics based on the simple and the most straightforward methods "can yield results which are almost surely optimal" for high-dimensional problems [12]. Following this observation, the term "blessing of dimensionality" was introduced [12,13]. It was clearly articulated as a basis of future data mining in the Donoho "Millenium manifesto" [14]. After that, the effects of the blessing of dimensionality were discovered in many applications, for example in face recognition [15], in analysis and separation of mixed data that lie on a union of multiple subspaces from their corrupted observations [16], in multidimensional cluster analysis [17], in learning large Gaussian mixtures [18], in correction of errors of multidimensonal machine learning systems [19], in evaluation of statistical parameters [20], and in the development of generalized principal component analysis that provides low-rank estimates of the natural parameters by projecting the saturated model parameters [21]....

Fractional Norms and Quasinorms Do Not Help to Overcome the Curse of Dimensionality

“…The “blessing of dimensionality” considers the same effect of concentration of distances from the different point of view [ 17 , 18 , 19 ]. The concentration of distances was discovered in the foundation of statistical physics and analysed further in the context of probability theory [ 20 , 21 ], functional analysis [ 22 ], and geometry (reviewed by [ 23 , 24 , 25 ]). The blessing of dimensionality allows us to use some specific high dimensional properties to solve problems [ 26 , 27 ].…”

Individual and Group Differences in Multivariate Domains: What Happens When the Number of Traits Increases?

“…Highdimensional spaces are not just large: they are vast and sparse in ways that stretch imagination, giving rise to a host of important but counterintuitive phenomena. These phenomena are usually discussed as part of the so-called "curse of dimensionality" (or its mirror image, the "blessing of dimensionality;" see Altman & Krzywinski, 2018;Gorban et al, 2020).…”

Biodiversity in NLP: modelling lexical meaning with the Fruit Fly Algorithm

“…As a starting point, we follow the desiderata highlighted in (Qasemizadeh, Kallmeyer, and Herbelot 2017) for a model of lexical learning: (A) high performance on fundamental semantic tasks, (B) efficiency, (C) low dimensionality for compact storage, (D) amenability to incremental learning, (E) interpretability. These considerations were not specifically brought up with cognitive plausibility in mind -it is for instance not so clear whether low dimensionality is a feature of cognition: see (Gorban, Makarov, and Tyukin 2020) for a discussion of the 'blessing of dimensionality'. Nevertheless, as we will see below, they point at crucial aspects of human language learning: the incremental process (D) that leads to the acquisition of full lexical competence (A), the robustness and efficiency of that process in spite of the poverty of the stimulus (B), and (to some extent) the ability of the speaker to formulate their linguistic knowledge in terms of explicit rules acting over categories (E).…”