A new estimator of intrinsic dimension based on the multipoint Morisita index

Abstract

The size of datasets has been increasing rapidly both in terms of number of variables and number of events. As a result, the empty space phenomenon and the curse of dimensionality complicate the extraction of useful information. But, in general, data lie on non-linear manifolds of much lower dimension than that of the spaces in which they are embedded. In many pattern recognition tasks, learning these manifolds is a key issue and it requires the knowledge of their true intrinsic dimension. This paper introduces a new estimator of intrinsic dimension based on the multipoint Morisita index. It is applied to both synthetic and real datasets of varying complexities and comparisons with other existing estimators are carried out. The proposed estimator turns out to be fairly robust to sample size and noise, unaffected by edge effects, able to handle large datasets and computationally efficient.

A New Approach for Interpreting the Morisita Index of Aggregation through Quadrat Size

“…The theory stated by Morisita [19] and further studies [18,54,55] do not specify how to define the range of the quadrat size or how to choose a quadrat size for I Mr . Golay et al [16] tackled the scale problem by linking I Mr index and the multifractality concept through quadrat-based methods, i.e., Rényi's generalized dimensions and the lacunarity index [16,17,56]. It appears that we are approaching the problem of scale by measuring the degree of crowding for different quadrat sizes.…”

Unsupervised feature selection based on the Morisita estimator of intrinsic dimension

“…The Morisita estimator of ID [38], M m , is derived from the multipoint Morisita index I m,δ [42][43][44]. I m,δ is computed by means of an E-dimensional grid of Q cells (or quadrats) of diagonal size δ superimposed over the data 4 points (see Figure 1).…”

“…In the remainder of this paper, the Morisita estimator of ID will be used only with m = 2 as advocated in [38]. The following steps summarize how to compute the ID of a data set using M m=2 : The procedure is illustrated in Figure 1 for E = 2.…”

“…In the next sections, M 2 will be computed using the MINDID algorithm [38] whose complexity is O(N * E * R). MINDID is part of the R package "IDmining" [45].…”

“…The present paper deals with a novel ID-based filter algorithm for RM. It relies on the recently introduced Morisita estimator of ID, M m , which was shown to be more effective than D 2 and df cor in situations where the data points were sparsely distributed [38]. Besides, the proposed algorithm fol- lows a SFS search strategy; it can process large and highly non-linear data, and its implementation is straightforward in R and Matlab.…”