Estimating Predictive Rate–Distortion Curves via Neural Variational Inference

Abstract

The Predictive Rate–Distortion curve quantifies the trade-off between compressing information about the past of a stochastic process and predicting its future accurately. Existing estimation methods for this curve work by clustering finite sequences of observations or by utilizing analytically known causal states. Neither type of approach scales to processes such as natural languages, which have large alphabets and long dependencies, and where the causal states are not known analytically. We describe Neural Predictive Rate–Distortion (NPRD), an estimation method that scales to such processes, leveraging the universal approximation capabilities of neural networks. Taking only time series data as input, the method computes a variational bound on the Predictive Rate–Distortion curve. We validate the method on processes where Predictive Rate–Distortion is analytically known. As an application, we provide bounds on the Predictive Rate–Distortion of natural language, improving on bounds provided by clustering sequences. Based on the results, we argue that the Predictive Rate–Distortion curve is more useful than the usual notion of statistical complexity for characterizing highly complex processes such as natural language.

Lossy‐Context Surprisal: An Information‐Theoretic Model of Memory Effects in Sentence Processing

“…Within rate-distortion theory, models that optimally allocate limited memory resources to best predict the future of a sequence have been studied, under the name of the Predictive Information Bottleneck (Still, 2014). The Predictive Information Bottleneck has recently been applied to study the resource requirements required for predicting linguistic sequences by Hahn and Futrell (2019).…”

“…We believe the most promising way to instantiate lossy-context surprisal in a broad coverage model is using RNNs with explicitly constrained memory capacity (e.g., using methods such as those developed by Alemi, Fischer, Dillon, & Murphy, 2017;Hahn & Futrell, 2019). Lossy-context surprisal predicts that such RNNs will yield surprisal values which are more predictive of human reading times than unconstrained RNNs (though any RNN is operating under at least mild memory constraints).…”

Approximating Information Measures for Fields

“…There is a hypothesis by Hilberg [ 16 ] that the excess entropy of natural language is infinite. This hypothesis can be partly confirmed by the original estimates of conditional entropy by Shannon [ 17 ], by the power-law decay of the estimates of the entropy rate given by the PPM compression algorithm [ 18 ], by the approximately power-law growth of vocabulary called Heaps’ or Herdan’s law [ 2 , 3 , 19 , 20 ], and by some other experiments applying neural statistical language models [ 21 , 22 ]. In parallel, Dębowski [ 1 , 2 , 3 ] supposed that the very large excess entropy in natural language may be caused by the fact that texts in natural language describe some relatively slowly evolving and very complex reality.…”

Information Theory and Language

“…Hahn, M. and Futrell, R., Estimating Predictive Rate–Distortion Curves via Neural Variational Inference [ 12 ].…”