Rate‐Distortion Theory

Abstract

Rate‐distortion theory is the branch of information theory that treats compressing the data produced by an information source down to a specified encoding rate that is strictly less than the source's entropy. This necessarily entails some lossiness, or distortion, between the original source data and the best approximation thereto that can be produced on the basis of the encoder's output bits. Rate‐distortion theory was introduced in the seminal works written in 1948 and 1959 by C. E. Shannon , the founder of information theory. We describe Shannon's contribution and then trace its subsequent development worldwide. Heavier than usual emphasis is placed on the concept of “matching” a channel to a source in the rate‐distortion sense, and also on the analogous matching of a source to a channel. Experimental evidence has been mounting in support of the hypothesis that living organisms often simultaneously achieve both of these matchings when processing their sensory inputs, thereby eliminating the need for the complex encoding and decoding operations that are needed in order to produce an information‐theoretically optimum system in the absence of such double matching.

Coarse-Grained Probabilistic Automata Mimicking Chaotic Systems

“…This is noteworthy: were the evolution driven only by the probabilistic diffusion, n t+1 = n t + σ t , the ǫ-entropy would have been ten times smaller, h diff (1/18, 252) ≃ 0.02. The effect of randomness is strongly enhanced by the deterministic evolution [21].…”

Entropy Approximation in Lossy Source Coding Problem

“…The above problem differs from the rate distortion theory [9,10], which is the most common approach to lossy source coding. Instead of specifying an upper limit on the allowed average distortion rate, an upper limit on the permitted distortion for any symbol is considered.…”

Opportunistic scheduling with limited channel state information: A rate distortion approach

“…The traditional rate-distortion problem [9] has been extended to bounds for Markov Sources in [10], [11], [12]. Additionally, researchers have considered the causal source coding problem due to its application to real-time processing.…”