Normalized Entropy Vectors, Network Information Theory and Convex Optimization

Abstract

Abstract-We introduce the notion of normalized entropicWhile, "in principle", it is possible to write down a characvectors-slightly different from the standard definition in the terization for the capacity region of most network information literature in that we normalize entropy by the logarithm of the theory problems, the difficulty is that this characterization is alphabet size. We argue that this definition is more natural for determining the capacity region of networks and, in particular, infinite-letter and non-convex. In other words, evaluating the that it smooths out the irregularities of the space of non-capacity region requires solving an infinite succession of nonnormalized entropy vectors and renders the closure of the convex optimization problems over certain distributions whose resulting space convex (and compact). Furthermore, the closure number of variables goes to infinity. This is in stark contrast of the space remains convex even under constraints imposed with point-to-point (single-user) memoryless channels where by memoryless channels internal to the network. It therefore follows that, for a large class of acyclic memoryless networks, the the characterizaton is both single-letter and convex.capacity region for an arbitrary set of sources and destinations can be found by maximization of a linear function over the S X convex set of channel-constrained normalized entropic vectors PXI5 (x15) and some linear constraints. While this may not necessarily make the problem simpler, it certainly circumvents the "infinite-letter characterization" issue, as well as the nonconvexity of earlier formulations, and exposes the core of the problem. We show that Fig. 1. A point-to-point communication problem. the approach allows one to obtain the classical cutset bounds via a duality argument. Furthermore, the approach readily shows that, To make this more explicit, consider the point-to-point for acyclic memoryless wired networks, one need only consider -the space of unconstrained normalized entropic vectors, thus memoryless channel of Fig. 1. The capacity is clearly separating channel and network coding-a result very recently C = max I(S; X) = max {H(X) -H(X S)}, (1) recognized in the literature. Ps ( ) Ps ( )

Scalable Capacity Bounding Models for Wireless Networks

“…While both [8] and [9] rely on the max-flow min-cut theorem [10] to show the converse (cf. upper bounding model), similar results on channel-network separation have been established in [11] by using normalized entropy vectors. For point-to-point channels, the same upper bounding models established in [1] have also been developed in [12] for DMCs with finite-alphabet, and in [13] under the notion of strong coordination, where total variation (i.e., an additive gap) is used to measure the difference between the desired joint distribution and the empirical joint distribution of a pair of sequences (or a pair of symbols as in empirical coordination).…”

On the entropy region of discrete and continuous random variables and network information theory

“…Γ * n denotes the space of non-normalized entropies. Theorem 1 (Convexity ofΩ * n ): The closure of the set of entropic vectors,Ω * n is convex [6]. and N y respectively, and let h x , h y ∈ Ω * n be the corresponding normalized entropy vectors.…”

“…Note that the validity of the proofs of Theorem 1 will not be affected when the underlying distributions satisfy some linear channel constraints. Therefore if we denote the space of entropic vectors that are constrained by the discrete memoryless channels in the network by Ω * n,c , we have [6], Theorem 2 (Channel-Constrained Entropic Vectors): Closure of the channel constrained entropic vectors,Ω * n,c , is convex.…”

“…In the networks for which separation of channel and network coding holds ( [9], [6]), channels affect the rate region only through their capacities and therefore the optimization over Ω * n,C in problem (14) can be replaced by an optimization over the unconstrained entropy region, Ω * n , and the set of capacity constraints. Therefore in the remaining we will focus on characterizing the unconstrained entropy region Ω * n .…”

MCMC methods for entropy optimization and nonlinear network coding

“…The set of all entropy vectors derived from n random variables is denoted by Γ * n and its closure,Γ * n is well known to be a convex cone. The entropy region is of great importance since maximizing the (weighted) throughput for a large class of wired acyclic networks can be reduced to convex optimization overΓ * n [2], [3]. Despite the importance attached to the entropy region, there exists very little in the way of explicitly characterizinḡ Γ * n for n ≥ 4 random variables.…”