Information Nonanticipative Rate Distortion Function and Its Applications

Abstract

This paper investigates applications of nonanticipative Rate Distortion Function (RDF) in a) zero-delay Joint Source-Channel Coding (JSCC) design based on average and excess distortion probability, b) in bounding the Optimal Performance Theoretically Attainable (OPTA) by noncausal and causal codes, and computing the Rate Loss (RL) of zero-delay and causal codes with respect to noncausal codes. These applications are described using two running examples, the Binary Symmetric Markov Source with parameter p, (BSMS(p)) and the multidimensional partially observed Gaussian-Markov source. For the multidimensional Gaussian-Markov source with square error distortion, the solution of the nonanticipative RDF is derived, its operational meaning using JSCC design via a noisy coding theorem is shown by providing the optimal encoding-decoding scheme over a vector Gaussian channel, and the RL of causal and zero-delay codes with respect to noncausal codes is computed. For the BSMS(p) with Hamming distortion, the solution of the nonanticipative RDF is derived, the RL of causal codes with respect to noncausal codes is computed, and an uncoded noisy coding theorem based on excess distortion probability is shown. The information nonanticipative RDF is shown to be equivalent to the nonanticipatory -entropy, which corresponds to the classical RDF with an additional causality or nonanticipative condition imposed on the optimal reproduction conditional distribution.

Optimal Estimation via Nonanticipative Rate Distortion Function and Applications to Time-Varying Gauss--Markov Processes

“…. Note that 1) is similar to the one derived in [15,Lemma IV.4]. Also, for 2) recall that a bounded and convex function is continuous.…”

“…The equivalence of the nonanticipatory -entropy, R ε 0,n (D), and NRDF, R na 0,n (D), is a direct consequence of the following equivalent characterization of conditional independence statements shown in [15].…”

Transfer Entropy in MDPs with Temporal Logic Specifications

“…The rest of the paper will deal with (6) V. OPTIMALITY CONDITIONS In this section, we derive a necessary optimality condition for (6). The result in this section generalizes [32], [33] to conditional directed information. In the following derivation, we assume β = 1 for simplicity.…”

Directed Information on Abstract Spaces: Properties and Variational Equalities

“…Comparing Theorem III.10, 1), with the lower semicontinuity of mutual information I(X n ; Y n ) ≡ I X n ;Y n (P X n , Q Y n |X n ), it is clear that directed information requires additional assumptions for its derivation (e.g., those given in Theorem III.5). Theorem III.5 together with Theorem III.10 are important to establish existence of the optimal reproduction distribution for the nonanticipative rate distortion functions defined by (I.7) [23], [42] (by utilizing Weierstrass' Theorem) and in general extremum problems of directed information involving minimization over − → Q 0,n (·|x n ) in some subset of M C2 1 (Y 0,n ). This is formally stated in the next theorem.…”

“…Generalized Information Nonanticipative or Sequential RDF. Consider the extremum problem of general information nonanticipative RDF, or sequential RDF [7], which is a variant of classical RDF [41], defined by [23], [42] R na (D) = lim sup n→∞ inf P Y i |Y i−1 ,X i , i=0,1,...,n ∈Q 0,n (D) 1 n + 1 I(X n → Y n ), (I. 7) where Q 0,n (D) is the fidelity constraint set defined by Q 0,n (D) = Q Y i |Y i−1 ,X i , i = 0, 1, .…”