The generalized ratios intrinsic dimension estimator

Abstract

Modern datasets are characterized by numerous features related by complex dependency structures. To deal with these data, dimensionality reduction techniques are essential. Many of these techniques rely on the concept of intrinsic dimension (), a measure of the complexity of the dataset. However, the estimation of this quantity is not trivial: often, the depends rather dramatically on the scale of the distances among data points. At short distances, the can be grossly overestimated due to the presence of noise, becoming smaller and approximately scale-independent only at large distances. An immediate approach to examining the scale dependence consists in decimating the dataset, which unavoidably induces non-negligible statistical errors at large scale. This article introduces a novel statistical method, , that allows estimating the as an explicit function of the scale without performing any decimation. Our approach is based on rigorous distributional results that enable the quantification of uncertainty of the estimates. Moreover, our method is simple and computationally efficient since it relies only on the distances among data points. Through simulation studies, we show that is asymptotically unbiased, provides comparable estimates to other state-of-the-art methods, and is more robust to short-scale noise than other likelihood-based approaches.

intRinsic: An R Package for Model-Based Estimation of the Intrinsic Dimension of a Dataset

“…In this paper, we illustrated intRinsic, an R package that implements novel routines for the ID estimation according to the models recently developed in Facco et al (2017); Allegra et al (2020); Denti et al (2022), andSantos-Fernandez et al (2022). intRinsic consists of a collection of high-level, user-friendly functions that, in turn, rely on efficient, low-level routines implemented in R and C++.…”

“…Maximum likelihood estimator. In a similar spirit, Denti et al (2022) took advantage of the distributional results in Equation 2 to derive a simple maximum likelihood estimator (MLE) and the corresponding confidence interval (CI). Trivially, the (unbiased) MLE for the shape parameter of a Pareto distribution is given by:…”

“…Bayesian estimator. It is also straightforward to derive an estimator according to a Bayesian perspective (Denti et al 2022), obtained with the specification of a prior distribution on the shape parameter d. The most natural prior to choose is a conjugate prior d ∼ Gamma (a, b). It is then immediate to derive the posterior distribution for the shape parameter:…”

“…The ratios of NNs of generic order are necessary when using the GRIDE model. See Appendix A and Denti et al (2022) for more details.…”

“…The package implements the two nearest neighbors (TWO-NN), the generalized ratio ID estimator (GRIDE), and the heterogeneous ID algorithm (HIDALGO) models, three state-of-theart ID estimators recently introduced in Facco, D 'Errico, Rodriguez, and Laio (2017); Denti, Doimo, Laio, and Mira (2022) and Allegra, Facco, Denti, Laio, and Mira (2020), respectively. These methods are likelihood-based estimators that rely on the theoretical properties of the distances among nearest neighbors.…”