Probing High-Order Dependencies With Information Theory

Abstract

Information theoretic measures (entropies, entropy rates, mutual information) are nowadays commonly used in statistical signal processing for real-world data analysis. The present work proposes the use of Auto Mutual Information (Mutual Information between subsets of the same signal) and entropy rate as powerful tools to assess refined dependencies of any order in signal temporal dynamics. Notably, it is shown how two-point Auto Mutual Information and entropy rate unveil information conveyed by higher order statistic and thus capture details of temporal dynamics that are overlooked by the (twopoint) correlation function. Statistical performance of relevant estimators for Auto Mutual Information and entropy rate are studied numerically, by means of Monte Carlo simulations, as functions of sample size, dependence structures and hyper parameters that enter their definition. Further, it is shown how Auto Mutual Information permits to discriminate between several different non Gaussian processes, having exactly the same marginal distribution and covariance function. Assessing higher order statistics via multipoint Auto Mutual Information is also shown to unveil the global dependence structure fo these processes, indicating that one of the non Gaussian actually has temporal dynamics that ressembles that of a Gaussian process with same covariance while the other does not.

Information Theory for Non-Stationary Processes with Stationary Increments

“…When k/T 1 m+1 is reduced, first the bias is positive and diminishes toward negative values and then converges to zero. This behavior was previously reported for the k-nn mutual information estimator applied for stationary processes [16,38,39], and we confirm it is valid for the fBm. We observed the same convergence for a large range of scales τ > 1: the ersatz entropy rate then converges to H 1 fBm + H ln τ for large T with the same behavior of the bias.…”

“…Fig. 4b shows that for a fixed window size T = 2 16 the ersatz entropy rate is proportional to H ln(τ). We have added a black line defined by the linear function H FBM 1 + H ln τ, as suggested by eq.…”

“…The right column of Fig. 3 shows the ersatz Shannon entropy and auto-mutual information for a fixed window size T = 2 16 when varying the scale parameter τ. The ersatz Shannon entropy behaves as (m − 1)H ln(τ), see Fig.…”

“…(1,τ) T depends only weakly on T, and seems to converge for larger T to the value H ln 1 , up to a small corrective term. 1 if X is a log-normal process of mean µ and standard deviation σ, then the process log X is Gaussian with the mean µ = log µ 2 √ µ 2 +σ 2 and the standard deviation σ = 2 log 1 + σ 2 µ 2 and the entropy of X can be expressed as [16]: Fig. 4 with m = 1) are reported in black for comparison.…”

Distance to Healthy Metabolic and Cardiovascular Dynamics From Fetal Heart Rate Scale-Dependent Features in Pregnant Sheep Model of Human Labor Predicts the Evolution of Acidemia and Cardiovascular Decompensation

“…We report in Table 5 the average fraction of missing data points in a time-window of size T =20 min: increasing the UCO strength is typically associated with an increase of missing data. Let's focus on the entropy rate h k (τ), which is algorithmicaly the most complex quantity: it has been reported that the bias of h k (τ) not only behaves as ( 39 , 40 ), similar to the bias of a sliding average over N points like m k (τ), but also that this bias is small. A time-window of 20 min should contain N = 20 × 60 × 4 = 4, 800 points, and even a reduction of 50% of available data points should leave more that 2,000 points so a bias smaller than 1% ( 40 ).…”