Empirical Estimation of Information Measures: A Literature Guide

Verdú, S.

doi:10.3390/e21080720

Cited by 53 publications

(47 citation statements)

References 111 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To analyze the relationship between the prediction errors of classical forecasting methods such as ARIMA, we build a feature space [ 18 ] based on Shannon entropy ( H ) features that presumably can be used to identify those TS instances where ARIMA forecasting errors are expected to be higher or lower, accordingly. These features are based on four entropy-based complexity measures, namely the frequentist binning approach (

) [ 22 ]; 2-Regimes (

) [ 19 ] and Permutation (

) entropy [ 23 ]. These three built upon the notions of symbolic dynamics, and the Spectral entropy (

) [ 24 ] based on the analysis of the spectrum of a time series.…”

Section: Methodsmentioning

confidence: 99%

Time Series Complexities and Their Relationship to Forecasting Performance

Ponce-Flores

Frausto–Solís

Santamaría-Bonfil

et al. 2020

Entropy

View full text Add to dashboard Cite

Entropy is a key concept in the characterization of uncertainty for any given signal, and its extensions such as Spectral Entropy and Permutation Entropy. They have been used to measure the complexity of time series. However, these measures are subject to the discretization employed to study the states of the system, and identifying the relationship between complexity measures and the expected performance of the four selected forecasting methods that participate in the M4 Competition. This relationship allows the decision, in advance, of which algorithm is adequate. Therefore, in this paper, we found the relationships between entropy-based complexity framework and the forecasting error of four selected methods (Smyl, Theta, ARIMA, and ETS). Moreover, we present a framework extension based on the Emergence, Self-Organization, and Complexity paradigm. The experimentation with both synthetic and M4 Competition time series show that the feature space induced by complexities, visually constrains the forecasting method performance to specific regions; where the logarithm of its metric error is poorer, the Complexity based on the emergence and self-organization is maximal.

show abstract

) [ 22 ]; 2-Regimes (

) [ 19 ] and Permutation (

) entropy [ 23 ]. These three built upon the notions of symbolic dynamics, and the Spectral entropy (

) [ 24 ] based on the analysis of the spectrum of a time series.…”

Section: Methodsmentioning

confidence: 99%

Time Series Complexities and Their Relationship to Forecasting Performance

Ponce-Flores

Frausto–Solís

Santamaría-Bonfil

et al. 2020

Entropy

View full text Add to dashboard Cite

show abstract

“…Multi-information as total dependence. Another important information measure that we will use in several ways is the multi-information for n variables (originally defined and called ''total correlation,'' by Watanabe (1960) and discussed and used by many others (Ting, 1962;Han, 1980)). It is defined as the difference between the sum of entropies of each variable separately and the joint entropy of all the variables together:…”

Section: Genetic Dependence Relationsmentioning

confidence: 99%

Toward an Information Theory of Quantitative Genetics

Galas

Kunert-Graf

Uechi

et al. 2021

Journal of Computational Biology

View full text Add to dashboard Cite

Quantitative genetics has evolved dramatically in the past century, and the proliferation of genetic data, in quantity as well as type, enables the characterization of complex interactions and mechanisms beyond the scope of its theoretical foundations. In this article, we argue that revisiting the framework for analysis is important and we begin to lay the foundations of an alternative formulation of quantitative genetics based on information theory. Information theory can provide sensitive and unbiased measures of statistical dependencies among variables, and it provides a natural mathematical language for an alternative view of quantitative genetics. In the previous work, we examined the information content of discrete functions and applied this approach and methods to the analysis of genetic data. In this article, we present a framework built around a set of relationships that both unifies the information measures for the discrete functions and uses them to express key quantitative genetic relationships. Information theory measures of variable interdependency are used to identify significant interactions, and a general approach is described for inferring functional relationships in genotype and phenotype data. We present information-based measures of the genetic quantities: penetrance, heritability, and degrees of statistical epistasis. Our scope here includes the consideration of both two- and three-variable dependencies and independently segregating variants, which captures additive effects, genetic interactions, and two-phenotype pleiotropy. This formalism and the theoretical approach naturally apply to higher multivariable interactions and complex dependencies, and can be adapted to account for population structure, linkage, and nonrandomly segregating markers. This article thus focuses on presenting the initial groundwork for a full formulation of quantitative genetics based on information theory.

show abstract

“…In general, joint entropy can be expressed in terms of conditional entropy

, as

where

and

denotes conditional probability. When the probability distributions are not known, it is not possible to calculate the exact value of

and it is necessary to calculate an estimator

from a sample [ 18 , 19 , 20 ].…”

Section: Preliminariesmentioning

confidence: 99%

Measuring Independence between Statistical Randomness Tests by Mutual Information

Karell-Albo

Legón-Pérez

Madarro-Capó

et al. 2020

Entropy

View full text Add to dashboard Cite

The analysis of independence between statistical randomness tests has had great attention in the literature recently. Dependency detection between statistical randomness tests allows one to discriminate statistical randomness tests that measure similar characteristics, and thus minimize the amount of statistical randomness tests that need to be used. In this work, a method for detecting statistical dependency by using mutual information is proposed. The main advantage of using mutual information is its ability to detect nonlinear correlations, which cannot be detected by the linear correlation coefficient used in previous work. This method analyzes the correlation between the battery tests of the National Institute of Standards and Technology, used as a standard in the evaluation of randomness. The results of the experiments show the existence of statistical dependencies between the tests that have not been previously detected.

show abstract

Empirical Estimation of Information Measures: A Literature Guide

Cited by 53 publications

References 111 publications

Time Series Complexities and Their Relationship to Forecasting Performance

Time Series Complexities and Their Relationship to Forecasting Performance

Toward an Information Theory of Quantitative Genetics

Measuring Independence between Statistical Randomness Tests by Mutual Information

Contact Info

Product

Resources

About