Universal estimation of directed information

Abstract

Abstract-Four estimators of the directed information rate between a pair of jointly stationary ergodic finite-alphabet processes are proposed, based on universal probability assignments. The first one is a Shannon-McMillan-Breiman type estimator, similar to those used by Verdú (2005) and Cai, Kulkarni, and Verdú (2006) for estimation of other information measures. We show the almost sure and L1 convergence properties of the estimator for any underlying universal probability assignment. The other three estimators map universal probability assignments to different functionals, each exhibiting relative merits such as smoothness, nonnegativity, and boundedness. We establish the consistency of these estimators in almost sure and L1 senses, and derive nearoptimal rates of convergence in the minimax sense under mild conditions. These estimators carry over directly to estimating other information measures of stationary ergodic finite-alphabet processes, such as entropy rate and mutual information rate, with near-optimal performance and provide alternatives to classical approaches in the existing literature. Guided by these theoretical results, the proposed estimators are implemented using the context-tree weighting algorithm as the universal probability assignment. Experiments on synthetic and real data are presented, demonstrating the potential of the proposed schemes in practice and the utility of directed information estimation in detecting and measuring causal influence and delay.

Recognizing Information Feature Variation: Message Importance Transfer Measure and Its Applications in Big Data

“…For example, the directed information [25][26][27][28] and Schreiber's transfer entropy [29] are commonly applied to infer the causality structure and characterize the information transfer process. Moreover, referring to the idea from dynamical system theory, new information transfer measures are proposed to indicate the causality between states and control the systems [30][31][32].…”

Universal estimation of directed information via sequential probability assignments

“…The estimatorÎ 1 is adapted from one universal divergence estimator in [14]. One disadvantage ofÎ 1 (X n → Y n ) is that it has a nonzero probability of being very large, which is overcome byÎ 2 , the estimator introduced in [20], by using information-theoretic functionals to "smooth" the estimate. Evidently we can show |Î 2 | ≤ log |Y|.…”

“…Based on parametric generalized linear model (GLM) assumption and stationary ergodic Markov assumption [19], they showed strong consistency results. Compared to [19], Zhao, Kim, Permuter, and Weissman [20] focused on universal methods and showed L 1 consistency for all jointly stationary ergodic process pairs with finite alphabet.…”

“…As an improvement and further development of [20], and a reflection of Jiao, Permuter, Zhao, Kim, and Weissman [21], the main contribution of this paper is a general framework for estimating information measures of stationary ergodic processes, using "single-letter" information-theoretic functionals. Although our methods can be applied in estimating a number of information measures, we focus-for concreteness and relevance to emerging applications-on estimating the directed information rate between a pair of jointly stationary ergodic processes.…”

Inferring causality in networks of WSS processes by pairwise estimation methods

“…It was pointed out in [24] that when two processes are jointly Gauss-Markov, there is an equivalence between Granger-causality and directed information. Equivalence relations between the two were derived in [4] for Gaussian linear models and in [5] under fairly general frameworks.…”