Estimation of Multivariate Shannon Entropy Using Moments

Abstract

Three new entropy estimators of multivariate distributions are introduced. The two cases considered here concern when the distribution is supported by a unit sphere and by a unit cube. In the former case, the consistency and the upper bound of the absolute error for the proposed entropy estimator are established. In the latter one, under the assumption that only the moments of the underlying distribution are available, a non-traditional estimator of the entropy is suggested. We also study the practical performances of the constructed estimators through simulation studies and compare the estimators based on the moment-recovered approaches with their counterparts derived by using the histogram and kth nearest neighbour constructions. In addition, one worked example is briefly discussed.

k-Nearest Neighbor Based Consistent Entropy Estimation for Hyperspherical Distributions

“…Another entropy estimator for hyperspherical data was developed recently by Mnatsakanov et al [10] using MR approach. We call this estimator the MR entropy estimator and denote it by H (MR) n (f ):…”

“…In Section 4, we present simulation studies using uniform hyperspherical distributions and aforementioned vMF probability models. In addition, the knn entropy estimator is compared with the MR approach proposed in Mnatsakanov et al [10]. We conclude this study in Section 5.…”

“…Recently, Mnatsakanov et al [10] proposed an entropy estimator for hyperspherical data based on the moment-recovery (MR) approach (see also Section 4.3).…”

Comprehensive review of association estimators for the inference of gene networks

“…These approaches directly estimate the MI by using the K-nearest neighborhood (KNN) method [98][99][100][101][102][103][104][105]. They compared those two approaches with the KNN entropy estimator [97,99,[106][107][108][109][110][111][112][113], which estimates the MI indirectly from the entropies by using yeast expression dataset with 6000 genes and 300 samples. The bias caused by the separately estimation of H(X) , H( Y ), and H( X , Y ) is decreased in the MI (1) and MI (2) methods.…”