Rate-distortion dimension of stochastic processes

Rezagah, Farideh Ebrahim; Jalali, Shirin; Erkip, Elza; Poor, H. Vincent

doi:10.1109/isit.2016.7541665

Cited by 6 publications

(3 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [9], the average RID of a block of samples with increasingly large block size is defined as the block-average information dimension (BID). It was later shown in [8] that BID coincides with RDD under certain conditions. The generalization in [26], however, relies on the log-scaling behavior of the entropy rate of the quantized samples.…”

Section: Related Workmentioning

confidence: 95%

“…It is shown that when the distortion tends to zero, the limiting value of the RDF is closely related to the differential entropy (if it exists) [3]. The compressibility notations are not limited to discrete and continuous sources: there has been some recent efforts to define this notion for sequences [4], [5], [6] and random processes [7], [8], [9], [10].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Compressibility Measures for Affinely Singular Random Vectors

Charusaie,

Amini,

Rini

2020

Preprint

View full text Add to dashboard Cite

There are several ways to measure the compressibility of a random measure; they include the general rate-distortion curve, as well as more specific notions such as Renyi information dimension (RID), and dimensional-rate bias (DRB). The RID parameter indicates the concentration of the measure around lower-dimensional subsets of the space while the DRB parameter specifies the compressibility of the distribution over these lowerdimensional subsets. While the evaluation of such compressibility parameters is well-studied for continuous and discrete measures (e.g., the DRB is closely related to the entropy and differential entropy in discrete and continuous cases, respectively), the case of discrete-continuous measures is quite subtle. In this paper, we focus on a class of multi-dimensional random measures that have singularities on affine lower-dimensional subsets. These cases are of interest when working with linear transformation of component-wise independent discrete-continuous random variables. Here, we evaluate the RID and DRB for such probability measures. We further provide an upper-bound for the RID of multi-dimensional random measures that are obtained by Lipschitz functions of component-wise independent discretecontinuous random variables (x). The upper-bound is shown to be achievable when the Lipschitz function is Ax, where A satisfies spark(A) = rank(A) + 1 (e.g., Vandermonde matrices). The above results in the case of discrete-domain moving-average processes with non-Gaussian excitation noise allow us to evaluate the block-average RID and to find a relationship between this parameter and other existing compressibility measures.

show abstract

Section: Related Workmentioning

confidence: 95%

Section: Introductionmentioning

confidence: 99%

Compressibility Measures for Affinely Singular Random Vectors

Charusaie,

Amini,

Rini

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Using a similar approach as in (Sefidgaran et al, 2022, Proof of Corollary 7), the generalization bound of Theorem 8 can be expressed in terms of the ratedistortion dimension of the optimization trajectories. Rezagah et al (2016) and Geiger and Koch (2019) have shown that for a large family of distortions ρpw, ŵq, the rate-distortion dimension of the process coincides with the Rényi information dimension of the process, and determines the fundamental limits of compressed sensing settings (Rényi, 1959;Jalali and Poor, 2014). The mentioned literature also provides practical methods that allow measuring efficiently the compressibility of the process.…”

Section: Dimension-based Boundsmentioning

confidence: 99%

Data-dependent Generalization Bounds via Variable-Size Compressibility

Sefidgaran¹,

Zaidi²

2023

Preprint

View full text Add to dashboard Cite

In this paper, we establish novel data-dependent upper bounds on the generalization error through the lens of a "variable-size compressibility" framework that we introduce newly here. In this framework, the generalization error of an algorithm is linked to a variable-size 'compression rate' of its input data. This is shown to yield bounds that depend on the empirical measure of the given input data at hand, rather than its unknown distribution. Our new generalization bounds that we establish are tail bounds, tail bounds on the expectation, and in-expectations bounds. Moreover, it is shown that our framework also allows to derive general bounds on any function of the input data and output hypothesis random variables. In particular, these general bounds are shown to subsume and possibly improve over several existing PAC-Bayes and data-dependent intrinsic dimension-based bounds that are recovered as special cases, thus unveiling a unifying character of our approach. For instance, a new data-dependent intrinsic dimension based bounds is established, which connects the generalization error to the optimization trajectories and reveals various interesting connections with rate-distortion dimension of process, Rényi information dimension of process, and metric mean dimension.

show abstract

On the information dimension rate of stochastic processes

Geiger

Koch

2017

2017 IEEE International Symposium on Information Theory (ISIT)

View full text Add to dashboard Cite

Abstract-Jalali and Poor ("Universal compressed sensing," arXiv:1406.7807v3, Jan. 2016) have recently proposed a generalization of Rényi's information dimension to stationary stochastic processes by defining the information dimension of the stochastic process as the information dimension of k samples divided by k in the limit as k ! 1. This paper proposes an alternative definition of information dimension as the entropy rate of the uniformlyquantized stochastic process divided by minus the logarithm of the quantizer step size 1/m in the limit as m ! 1. It is demonstrated that both definitions are equivalent for stochastic processes that are ⇤ -mixing, but that they may differ in general. In particular, it is shown that for Gaussian processes with essentially-bounded power spectral density (PSD), the proposed information dimension equals the Lebesgue measure of the PSD's support. This is in stark contrast to the information dimension proposed by Jalali and Poor, which is 1 if the process's PSD is positive on a set of positive Lebesgue measure, irrespective of its support size.

show abstract

Rate-distortion dimension of stochastic processes

Cited by 6 publications

References 19 publications

Compressibility Measures for Affinely Singular Random Vectors

Compressibility Measures for Affinely Singular Random Vectors

Data-dependent Generalization Bounds via Variable-Size Compressibility

On the information dimension rate of stochastic processes

Contact Info

Product

Resources

About