On the Information Dimension of Stochastic Processes

Abstract

In 1959, Rényi proposed the information dimension and the d-dimensional entropy to measure the information content of general random variables. This paper proposes a generalization of information dimension to stochastic processes by defining the information dimension rate as the entropy rate of the uniformly-quantized stochastic process divided by minus the logarithm of the quantizer step size 1/m in the limit as m → ∞. It is demonstrated that the information dimension rate coincides with the rate-distortion dimension, defined as twice the rate-distortion function R(D) of the stochastic process divided by − log D in the limit as D ↓ 0. It is further shown that, among all multivariate stationary process with a given (matrix-valued) spectral distribution function (SDF), the Gaussian process has the largest information dimension rate, and that the information dimension rate of multivariate stationary Gaussian processes is given by the average rank of the derivative of the SDF. The presented results reveal that the fundamental limits of almost zero-distortion recovery via compressible signal pursuit and almost lossless analog compression are different in general. Index TermsInformation dimension, Gaussian process, rate-distortion dimension dimension rate d({X t }) is equal to twice the PSD's bandwidth. This is consistent with the intuition that for such processes not all samples contain information. For example, if the bandwidth of the PSD is 1/4, then we expect that half of the samples in {X t } can be expressed as linear combinations of the other samples and, hence, do not contain information. In contrast, we show that the information dimension d ′ ({X t }) is one if F ′ X is positive on any set with positive Lebesgue measure. In other words, d ′ ({X t }) does not capture the dependence of the information dimension on the support size of F ′ X . By emulating the proof of [2, Lemma 3.2], we further show that for, any stochastic process {X t }, the informationThe rest of this paper is organized as follows. In Section II, we introduce the notation used in this paper. In Section III, we present preliminary as well as novel results on the Rényi information dimension of RVs and random vectors. In Section IV, we present our definition of the information dimension rate of a stochastic process, discuss its connection to the rate-distortion dimension, and compute the information dimension rate of stationary Gaussian processes. In Section V, we review the information dimension proposed by Jalali and Poor and discuss its relation to d({X t }). In Section VI, we briefly discuss the operational meanings of information dimension in compressed sensing and zero-distortion recovery. Section VII concludes the paper with a discussion of the obtained results.Some of the proofs are deferred to the appendix. II. NOTATION AND PRELIMINARIESWe denote by R, C, and Z the set of real numbers, the set of complex numbers, and the set of integers, respectively.We further denote by R + and N the set of nonnegative real numbers and the set of posit...

Metric mean dimension and analog compression

“…It is given in terms of the mean box dimension of the set S and it is deduced from a nite-dimensional embedding theorem involving the upper box-counting (Minkowski) dimension (see [22]). We obtain also lower bounds on compression rates for a xed stationary process in terms of the rate-distortion dimension (see [6]; see also [9] for a more detailed treatment on the connections between various notions of dimension for stochastic processes).…”

“…Theorem VIII.5 is stronger than (15). Indeed, in general dim R,2 ≤ d 0 (µ) (see [9,Theorem 14]) and the equality can be strict (see [41,Example 1]). Also, ψ * -mixing is a quite restrictive assumption.…”

“…Overview. In recent years, the theory of compression for analog sources (modeled by realvalued stochastic processes) underwent a major development (as a sample of such results see [2], [3], [4], [5], [6], [7], [8], [9]). There are two key dierences with the classical Shannon's model of compression for discrete sources.…”

Compressibility Measures for Affinely Singular Random Vectors

“…The extension of RID to discrete-domain stochastic processes was considered in both [9] and [26]. In [9], the average RID of a block of samples with increasingly large block size is defined as the block-average information dimension (BID).…”

A Unified Discretization Approach to Compute-Forward: From Discrete to Continuous Inputs

“…Note that, in addition to the existence of d(u 1 |Y ) and d(u 2 |Y, u 1 ), the finiteness assumption on I(u 1 , u 2 ; Y ) is essential, as evidenced by Example 1 in[49] 5. This can be justified by the fact that Fq is isomorphic to ( q , ⊕q, ⊗q) or Z/qZ, which live on finite (and thus discrete) subsets of the real field R.October 4, 2021 DRAFT…”