A pragmatic entropy and differential entropy estimator for small datasets

Abstract

Abstract-A pragmatic approach for entropy estimation is presented, first for discrete variables, then in the form of an extension for handling continuous and/or multivariate ones. It is based on coincidence detection, and its application leads to algorithms with three main attractive features: they are easy to use, can be employed without any a priori knowledge concerning source distribution (not even the alphabet cardinality K of discrete sources) and can provide useful estimates even when the number of samples, N , is less than K, for discrete variables, whereas plug-in methods typically demand N >> K for a proper approximation of probability mass functions. Experiments done with both discrete and continuous random variables illustrate the simplicity of use of the proposed method, whereas numerical comparisons to other methods show that, in spite of its simplicity, useful results are yielded.

On the minimum probability of classification error through effective cardinality comparison

“…continuous L−dimensional multivariate data coupled to categorical data) they should be separately preprocessed for proper definition of multivariate coincidence event. As proposed in [19], in the Section entitled Setting and testing a coincidence neighbourhood, if the standard deviations in each dimension of the continuous data are normalized, one may take advantage of an approximative relationship between a first guess for the effective cardinality and the hyper-volume ∆. For instance, given a first rough guess for effective cardinality of about C (2) = 1000, and L = 3, then one may set ∆ ≈…”

Bias-Compensated Estimator for Intrinsic Dimension and Differential Entropy: A Visual Multiscale Approach

“…Besides, [26] presents entropy as an effective cardinality in logarithmic scale. Likewise, DE can be regarded as an effective volume (in logarithmic scale) [20], [25].…”

“…On the other hand, in [19], the problem of DE estimation in high-dimensional spaces was tackled through a simple but data-efficient approach, referred to as the Coincidence Method (CM), originally applied in Physics. In [20] this method was extended to differential entropy estimation in the pattern recognition context, which clearly shows that the correlation dimension in [14] uses the same empirical coincidence ratio as the entropy estimation method proposed in [19].…”

Statistical estimation of conditional Shannon entropy

“…Since Φ(−z) ≤ 1 √ 2πz e − z 2 2 for z > 1, taking into account inequalities (26), (28) and (35) we get, for k = k n ∝ n α , x ∈ B 1,n and all n large enough,…”

“…Note that the probability distribution discretization techniques for a random variable having a density (w.r.t. the Lebesgue measure) and evaluation of the Shannon entropy for thus arising random variables do not lead to the differential entropy as the mash of the discretization tends to zero (see, e.g., Theorem 8.3.1 in [13] and [28]). More generally, when a measure σ is fixed on a measure space (S, B), one can define the notion of the entropy of a probability measure ν given on the same space and absolutely continuous w.r.t.…”