An approximation to the distribution of finite sample size mutual information estimates

Goebel, Bernhard; Dawy, Zaher; Hagenauer, J.; Mueller, Jakob C.

doi:10.1109/icc.2005.1494518

Cited by 59 publications

(68 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nonetheless, a less computationally expensive method can be presented, without introducing very strong assumptions. It has been shown that the mutual information between independent random variables (X & Y ), when estimated from relative frequencies, follows a very good approximation of a gamma distribution with parameters a = (|X| − 1)(|Y | − 1)/2 and b = 1/(N ln 2) [70,71]:…”

Section: Methodsmentioning

confidence: 99%

Information-theoretic approach to lead-lag effect on financial markets

Fiedor

2014

Eur. Phys. J. B

View full text Add to dashboard Cite

Abstract.Recently the interest of researchers has shifted from the analysis of synchronous relationships of financial instruments to the analysis of more meaningful asynchronous relationships. Both types of analysis are concentrated mostly on Pearson's correlation coefficient and consequently intraday lead-lag relationships (where one of the variables in a pair is time-lagged) are also associated with them. Under the Efficient-Market Hypothesis such relationships are not possible as all information is embedded in the prices, but in real markets we find such dependencies. In this paper we analyse lead-lag relationships of financial instruments and extend known methodology by using mutual information instead of Pearson's correlation coefficient. Mutual information is not only a more general measure, sensitive to non-linear dependencies, but also can lead to a simpler procedure of statistical validation of links between financial instruments. We analyse lagged relationships using New York Stock Exchange 100 data not only on an intraday level, but also for daily stock returns, which have usually been ignored.

show abstract

Section: Methodsmentioning

confidence: 99%

Information-theoretic approach to lead-lag effect on financial markets

Fiedor

2014

Eur. Phys. J. B

View full text Add to dashboard Cite

show abstract

“…Hence, knowledge of the distribution of the estimator is required in order to establish confidence bounds on estimates for a given sample, and to determine the critical values of the estimator. However, unlike the linear correlation coefficient, where the distribution of a sample-estimate follows a t-distribution, an equivalent analytical expression for f (Î) cannot be derived for the expression in (2) [5].…”

Section: B Distribution Of Mutual Informationmentioning

confidence: 99%

“…Alternatively, expressions for computing the mean, variance and conditional distribution p(I|n) have been derived using the assumption of a prior distribution over the point density estimates [2], [4]. An expression for the distribution of MI has also been described based on a second-order Taylor series expansion of the discrete MI estimator [5].…”

Section: B Distribution Of Mutual Informationmentioning

confidence: 99%

See 1 more Smart Citation

Critical Values of a Kernel Density-based Mutual Information Estimator

May

Dandy

Maier

et al. 2006

The 2006 IEEE International Joint Conference on Neural Network Proceedings

View full text Add to dashboard Cite

Abstract-Recently, mutual information (MI) has become widely recognized as a statistical measure of dependence that is suitable for applications where data are non-Gaussian, or where the dependency between variables is non-linear. However, a significant disadvantage of this measure is the inability to define an analytical expression for the distribution of MI estimators, which are based upon a finite dataset. This paper deals specifically with a popular kernel density based estimator, for which the distribution is determined empirically using Monte Carlo simulation. The application of the critical values of MI derived from this distribution to a test for independence is demonstrated within the context of a benchmark input variable selection problem.

show abstract

“…Analytical expressions of the distribution of MI estimates are given from a MI Taylor expansion in terms of the anomalies of the estimated probabilities [27,37]. Here, we adopt a different approach by considering anomalies of the estimated expectations.…”

Section: Bias Variance Quantiles and Distribution Of MI Estimation mentioning

confidence: 99%

Minimum Mutual Information and Non-Gaussianity through the Maximum Entropy Method: Estimation from Finite Samples

Pires

Perdigão

2013

Entropy

View full text Add to dashboard Cite

Abstract:The Minimum Mutual Information (MinMI) Principle provides the least committed, maximum-joint-entropy (ME) inferential law that is compatible with prescribed marginal distributions and empirical cross constraints. Here, we estimate MI bounds (the MinMI values) generated by constraining sets T cr comprehended by m cr linear and/or nonlinear joint expectations, computed from samples of N iid outcomes. Marginals (and their entropy) are imposed by single morphisms of the original random variables.N-asymptotic formulas are given both for the distribution of cross expectation's estimation errors, the MinMI estimation bias, its variance and distribution. A growing T cr leads to an increasing MinMI, converging eventually to the total MI. Under N-sized samples, the MinMI increment relative to two encapsulated sets T cr1  T cr2 (with numbers of constraints As an example, we have set marginals to being normally distributed (Gaussian) and have built a sequence of MI bounds, associated to successive non-linear correlations due to joint non-Gaussianity. Noting that in real-world situations available sample sizes can be rather low, the relationship between MinMI bias, probability density over-fitting and outliers is put in evidence for under-sampled data. OPEN ACCESSEntropy 2013, 15 722

show abstract

An approximation to the distribution of finite sample size mutual information estimates

Cited by 59 publications

References 8 publications

Information-theoretic approach to lead-lag effect on financial markets

Information-theoretic approach to lead-lag effect on financial markets

Critical Values of a Kernel Density-based Mutual Information Estimator

Minimum Mutual Information and Non-Gaussianity through the Maximum Entropy Method: Estimation from Finite Samples

Contact Info

Product

Resources

About