The Bidirectional Communication Theory--A Generalization of Information Theory

Marko, Hans

doi:10.1109/tcom.1973.1091610

Cited by 230 publications

(224 citation statements)

References 9 publications

Supporting

Mentioning

219

Contrasting

Unclassified

Order By: Relevance

“…Directed information theory measures are needed to appropriately represent the sequential availability of side information in these settings. The causally conditioned probability of X given Y from the MarkoMassey theory of directed information [6,7] is:…”

Section: Side Information and Directed Information Theorymentioning

confidence: 99%

“…with the increase in growth rate by having causally available side information being the directed information [6,7] …”

Section: Theorem 3 (From Permuter Et Al [4])mentioning

confidence: 99%

“…The Kelly criterion, as the resulting investment allocation is known, and related Kellyinspired criteria are commonly employed for investment and gambling. Permuter et al's recent work [4] makes the ideas of Kelly investment applicable to more complex settings with sequentially inter-dependent event outcomes P(X||Y) = ∏ T t=1 P(X t |X 1:t−1 ,Y t ) for event outcome sequences X and side information sequences Y using the Marko-Massey theory of directed information [6,7].…”

Section: Introductionmentioning

confidence: 99%

“…The maximum entropy distributions for joint, conditional, and marginal distributions with moment-matching constraints correspond to many state-of-the-art machine learning techniques, including Markov random fields and conditional random fields [8]. The principle of maximum causal entropy [9] extends the maximum entropy approach to settings with information revelation and feedback using directed information theory [6,7] with applications to single [9] and multi-agent decision prediction [10].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Process-conditioned investing with incomplete information using maximum causal entropy

Ziebart

2012

AIP Conference Proceedings

View full text Add to dashboard Cite

Abstract. Investing to optimally maximize the growth rate of wealth based on sequences of event outcomes has many information-theoretic interpretations. Namely, the mutual information characterizes the benefit of additional side information being available when making investment decisions [1] in settings where the probabilistic relationships between side information and event outcomes are known. Additionally, the relative variant of the principle of maximum entropy [2] provides the optimal investment allocation in the more general setting where the relationships between side information and event outcomes are only partially known [3]. In this paper, we build upon recent work characterizing the growth rates of investment in settings with inter-dependent side information and event outcome sequences [4]. We consider the extension to settings with inter-dependent event outcomes and side information where the probabilistic relationships between side information and event outcomes are only partially known. We introduce the principle of minimum relative causal entropy to obtain the optimal worst-case investment allocations for this setting. We present efficient algorithms for obtaining these investment allocations using convex optimization techniques and dynamic programming that illustrates a close connection to optimal control theory.

show abstract

Section: Side Information and Directed Information Theorymentioning

confidence: 99%

“…with the increase in growth rate by having causally available side information being the directed information [6,7] …”

Section: Theorem 3 (From Permuter Et Al [4])mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Process-conditioned investing with incomplete information using maximum causal entropy

Ziebart

2012

AIP Conference Proceedings

View full text Add to dashboard Cite

show abstract

“…IRST introduced by Marko [1] and Massey [2], directed information arises as a natural counterpart of mutual information for channel capacity when causal feedback from the receiver to the sender is present. In [3] and [4], Kramer extended the use of directed information to discrete memoryless networks with feedback, including the two-way channel and the multiple access channel.…”

mentioning

confidence: 99%

Universal estimation of directed information

Zhao

Permuter

Kim

et al. 2010

2010 IEEE International Symposium on Information Theory

View full text Add to dashboard Cite

Abstract-Four estimators of the directed information rate between a pair of jointly stationary ergodic finite-alphabet processes are proposed, based on universal probability assignments. The first one is a Shannon-McMillan-Breiman type estimator, similar to those used by Verdú (2005) and Cai, Kulkarni, and Verdú (2006) for estimation of other information measures. We show the almost sure and L1 convergence properties of the estimator for any underlying universal probability assignment. The other three estimators map universal probability assignments to different functionals, each exhibiting relative merits such as smoothness, nonnegativity, and boundedness. We establish the consistency of these estimators in almost sure and L1 senses, and derive nearoptimal rates of convergence in the minimax sense under mild conditions. These estimators carry over directly to estimating other information measures of stationary ergodic finite-alphabet processes, such as entropy rate and mutual information rate, with near-optimal performance and provide alternatives to classical approaches in the existing literature. Guided by these theoretical results, the proposed estimators are implemented using the context-tree weighting algorithm as the universal probability assignment. Experiments on synthetic and real data are presented, demonstrating the potential of the proposed schemes in practice and the utility of directed information estimation in detecting and measuring causal influence and delay.

show abstract