Improvement of the k-nn Entropy Estimator with Applications in Systems Biology

Abstract

Abstract:In this paper, we investigate efficient estimation of differential entropy for multivariate random variables. We propose bias correction for the nearest neighbor estimator, which yields more accurate results in higher dimensions. In order to demonstrate the accuracy of the improvement, we calculated the corrected estimator for several families of random variables. For multivariate distributions, we considered the case of independent marginals and the dependence structure between the marginal distributions described by Gaussian copula. The presented solution may be particularly useful for high dimensional data, like those analyzed in the systems biology field. To illustrate such an application, we exploit differential entropy to define the robustness of biochemical kinetic models.

K-th Nearest Neighbor (KNN) Entropy Estimates of Complexity and Integration from Ongoing and Stimulus-Evoked Electroencephalographic (EEG) Recordings of the Human Brain

“…A major disadvantage of using KNN entropy estimators to estimate EEG integration and complexity is that even though they are relatively efficient to compute for small multivariate data spaces [ 26 , 29 ], they are computationally intensive to compute for the large data spaces typical of EEG recordings. This is mainly due to the need to perform a SVD for each time point of an EEG epoch as part of the G-KNN computation (Equation (6)) of the joint entropy, H(X) , and conditional entropy, H(X i |X–X i ), and the need to compute the conditional entropy multiple times during the computation of C I (X) (see Equation (2)).…”

“…Fortunately, certain entropy estimators are minimally affected by the nonstationarity of a signal. One such estimator is the nonparametric k-th nearest neighbor (KNN) entropy estimator [ 26 , 27 , 28 , 29 , 30 ], which was used here to compute the entropies composing I(X) and C I (X) . Here, the entropy of a data space is estimated by approximating the multivariate probability density function, f X (X) , at each point, x i , in the space in terms of the ratio of the fraction of other points in a data point’s neighborhood to the volume of its neighborhood [ 29 ]: where N p is the number of data points in the space, Vol i is the volume of the neighborhood, and k(x i ) is the number of points in the neighborhood of x i other than x i .…”

“…Moreover, KNN entropy estimators are nonparametric and thus avoid entropy distortions that may arise when using an estimator derived via specific distributional assumptions of the data that are inaccurate, as was recently observed when a Gaussian entropy estimator was used to compute the complexity of non-Gaussian EEG data [ 11 ]. Finally, KNN entropy estimators are easy to implement and efficient to compute for multidimensional variable spaces [ 26 , 29 ], such as a set of EEG signals (although there are still limits to this efficiency for large spaces; see Section 4.3 , below).…”

“…The specific procedure for computing K G-KNN (X) may be found in [ 29 ]. Note that in the case of d = 1, log(σ l i /σ 1 i ) = log(σ 1> i /σ 1 i ) = 0, and Equation (6) reduces to a variant of the Kraskov-Stögbauer-Grassberger (KSG) KNN entropy estimator [ 26 , 28 ]: …”

Statistical Estimation of the Shannon Entropy

“…J.Ma and Z.Sun [17] analyzed the copula entropy estimation to get the mutual information estimate. The Gaussian copula in the entropy estimation is also used in [7]. A nonparanormal information estimation is considered in [23].…”