Detection of low dimensionality and data denoising via set estimation techniques

Abstract

This work is closely related to the theories of set estimation and manifold estimation. Our object of interest is a, possibly lower-dimensional, compact set S ⊂ R d . The general aim is to identify (via stochastic procedures) some qualitative or quantitative features of S, of geometric or topological character. The available information is just a random sample of points drawn on S. The term "to identify" means here to achieve a correct answer almost surely (a.s.) when the sample size tends to infinity. More specifically the paper aims at giving some partial answers to the following questions: is S full dimensional? Is S "close to a lower dimensional set" M? If so, can we estimate M or some functionals of M (in particular, the Minkowski content of M)? As an important auxiliary tool in the answers of these questions, a denoising procedure is proposed in order to partially remove the noise in the original data. The theoretical results are complemented with some simulations and graphical illustrations. arXiv:1702.05193v2 [math.ST] 3 Nov 2017 Estimation of some other relevant quantities in a manifold, Niyogi, Smale and Weinberger (2008), Chen and Müller (2012). Dimensionality reduction, Genovese et al. (2012a), Tenebaum et al. (2000).The problems under study. The contents of the paper. We are interested in getting some information (in particular, regarding dimensionality and Minkowski content) about a compact set M ⊂ R d . While the set M is typically unknown, we are supposed to have a random sample of points X 1 , . . . , X n whose distribution P X has a support "close to M". To be more specific, we consider two different models:The noiseless model : the support of P X is M itself; Aamari and Levrard (2015), Amenta et al. (2002), Cholaquidis et al. (2014), Cuevas and Fraiman (1997). The parallel (noisy) model : the support of P X is the parallel set S of points within a distance to M smaller than R 1 , for some R 1 > 0, where M is a d -dimensional set and d ≤ d; Berrendero et al. (2014). Note that other different models "with noise" are considered in Genovese et al. (2012a), Genovese et al. (2012b) and Genovese et al (2012c).In Section 3 we first develop, under the noiseless model, an algorithmic procedure to identify, eventually, almost surely (a.s.), whether or not M has an empty interior; this is achieved in Theorems 1 and 2 below. A positive answer would essentially entail (under some conditions, see the beginning of Section 3) that M has a dimension smaller than that of the ambient space.Then, assuming the noisy model andM = ∅ ( whereM denotes the interior of M) Theorems 3 (i) and 4 (i) provide two methods for the estimation of the maximum level of noise R 1 , giving also the corresponding convergence rates. If R 1 is known in advance, the remaining results in Theorems 3 and 4 allow us also to decide whether or not the "inside set" M has an empty interior.The identification methods are "algorithmic" in the sense that they are based on automatic procedures to perform them with arbitrary precision. This will requi...

On boundary detection

“…that the dimension of the support is lower than the dimension of the ambient space, can not be replaced by a noisy model in which the support is "around" a lower dimensional manifold. However, in such a case, performing a preliminary manifold estimation before running our test (see for instance Genovese, et al (2012) or Aaron et al (2017)) can be used to overcome this problem. Even if the manifold estimator is not a d ′ -dimensional manifold, we may expect that by imposing stronger conditions on the sequence k n , our approach can work.…”

“…However, as is shown in Figure 4, sometimes this gives "too many" boundary observations (as in the half sphere) and sometimes "too few"(as in the Möbius ring). To overcome this, we will adapt, using tangent spaces, the method given in Aaron et al (2017) to detect "boundary balls".…”

“…In Aaron et al (2017), M is d-dimensional and boundary observations are identified as those with large Voronoi cells (recall that Vor(…”

Surface and length estimation based on Crofton's formula

“…Proposition 1 in Aaron, Cholaquidis and Cuevas (2017) proves that a Borel set that satisfies the inside rolling condition is standard w.r.t. any measure ν with density f (w.r.t.…”

Universally consistent estimation of the reach

“…Recently, the study of statistical methods for manifold valued data (known as manifold learning) has gained attention, due to its application in dimension reduction (among others). The aim is to recover a lower dimensional structure from the data, see for instance Genovese et al (2012a,b); Fefferman et al (2016); Niyogi et al (2008) and the references therein, or a functional of it, see for instance Aaron, et al (2017); Aaron and Cholaquidis (2020). Several classical problems have been tackled in this setting, such as density estimation.…”