Algorithms for the Communication of Samples

Theis, Lucas; Yosri, Noureldin

doi:10.48550/arxiv.2110.12805

Cited by 2 publications

(12 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The use of quantization continues to cause a mismatch between training and test time performance, and how much this affects compression performance is still not clearly understood. Reverse channel coding is a promising alternative which eliminates the need for quantization, but has only recently been considered in neural compression [116]. Open questions include the design of efficient coding schemes and the impact these schemes have on performance when compared to approaches based on quantization.…”

Section: Discussion and Open Problemsmentioning

confidence: 99%

“…One is a simple and efficient approach for simulating channels with additive uniform noise, and one is a general approach for communicating samples of arbitrary distributions. For a more thorough introduction to reverse channel coding, see Theis & Yosri [116].…”

Section: Compression Without Quantizationmentioning

confidence: 99%

“…While dithered quantization can be computationally and statistically efficient, it is only able to communicate certain simple distributions. Several general algorithms have been developed to communicate a sample from arbitrary distributions [119][120] [121][114][122] [116]. Here we are describing one algorithm based on importance sampling.…”

Section: Minimal Random Codingmentioning

confidence: 99%

“…Havasi et al [122] applied minimal random coding to model compression, while Flamich et al [115] used it for image compression. Theis & Yosri [116] showed that the coding cost of minimal random coding can be further reduced without any loss in quality.…”

Section: Minimal Random Codingmentioning

confidence: 99%

See 3 more Smart Citations

An Introduction to Neural Data Compression

Yang¹,

Mandt²,

Theis³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Neural compression is the application of neural networks and other machine learning methods to data compression. While machine learning deals with many concepts closely related to compression, entering the field of neural compression can be difficult due to its reliance on information theory, perceptual metrics, and other knowledge specific to the field. This introduction hopes to fill in the necessary background by reviewing basic coding topics such as entropy coding and rate-distortion theory, related machine learning ideas such as bits-back coding and perceptual metrics, and providing a guide through the representative works in the literature so far.

show abstract

Section: Discussion and Open Problemsmentioning

confidence: 99%

Section: Compression Without Quantizationmentioning

confidence: 99%

Section: Minimal Random Codingmentioning

confidence: 99%

Section: Minimal Random Codingmentioning

confidence: 99%

See 2 more Smart Citations

An Introduction to Neural Data Compression

Yang¹,

Mandt²,

Theis³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…For example, in image compression, where we assume each X i ∈ R m to be a single image realization, Y ⊆ R m . Even for 8-bit grayscale images, full precision quantization would require 2 8 • m points, and although one could provide better discretization schemes, they may still require an intractable number of points.…”

Section: A Blahut-arimoto Fails To Scalementioning

confidence: 99%

Neural Estimation of the Rate-Distortion Function With Applications to Operational Source Coding

Lei¹,

Hassani²,

Bidokhti³

2022

Preprint

View full text Add to dashboard Cite

A fundamental question in designing lossy data compression schemes is how well one can do in comparison with the ratedistortion function, which describes the known theoretical limits of lossy compression. Motivated by the empirical success of deep neural network (DNN) compressors on large, real-world data, we investigate methods to estimate the rate-distortion function on such data, which would allow comparison of DNN compressors with optimality. While one could use the empirical distribution of the data and apply the Blahut-Arimoto algorithm, this approach presents several computational challenges and inaccuracies when the datasets are large and high-dimensional, such as the case of modern image datasets. Instead, we re-formulate the rate-distortion objective, and solve the resulting functional optimization problem using neural networks. We apply the resulting rate-distortion estimator, called NERD, on popular image datasets, and provide evidence that NERD can accurately estimate the rate-distortion function. Using our estimate, we show that the rate-distortion achievable by DNN compressors are within several bits of the rate-distortion function for real-world datasets. Additionally, NERD provides access to the rate-distortion achieving channel, as well as samples from its output marginal. Therefore, using recent results in reverse channel coding, we describe how NERD can be used to construct an operational one-shot lossy compression scheme with guarantees on the achievable rate and distortion. Experimental results demonstrate competitive performance with DNN compressors. Index TermsGenerative models, lossy compression, neural networks, rate-distortion theory, reverse channel coding I. INTRODUCTIONDriven by advances in deep neural network (DNN) compression schemes, rapid progress has been made in finding highperforming lossy compression schemes for large, high-dimensional datasets that remain practical [1]- [4]. While these methods have empirically shown to outperform classical compression schemes for real-world data (e.g. images), it remains unknown as to how well they perform in comparison to the fundamental limit, which is given by the rate-distortion function. To investigate this question, one approach is to examine a stylized data source with a known probability distribution that is analytically tractable, such as the sawbridge random process, as done in [5]. This allows for a closed-form solution of the rate-distortion function; one can then compare it with empirically achievable rate and distortion of DNN compressors trained on realizations of the source. However, this approach does not evaluate DNN compressors on true sources of interest, such as real-world images, for which architectural choices such as convolutional layers have been engineered [6]. Thus, evaluating the rate-distortion function on these sources is paramount to understanding the efficacy of DNN compressors on real-world data.Furthermore, a class of information-theoretically designed one-shot lossy source codes with near-optimal rate-dist...

show abstract

Algorithms for the Communication of Samples

Cited by 2 publications

References 22 publications

An Introduction to Neural Data Compression

An Introduction to Neural Data Compression

Neural Estimation of the Rate-Distortion Function With Applications to Operational Source Coding

Contact Info

Product

Resources

About