The sequence memoizer

Abstract

Probabilistic models of sequences play a central role in most machine translation, automated speech recognition, lossless compression, spell-checking, and gene identification applications to name but a few. Unfortunately, real-world sequence data often exhibit long range dependencies which can only be captured by computationally challenging, complex models. Sequence data arising from natural processes also often exhibits power-law properties, yet common sequence models do not capture such properties. The sequence memoizer is a new hierarchical Bayesian model for discrete sequence data that captures long range dependencies and power-law characteristics, while remaining computationally attractive. Its utility as a language model and general purpose lossless compressor is demonstrated.

Perplexity on Reduced Corpora

“…For example, promising research directions include using a general law such as the Zipf-Mandelbrot law (Mandelbrot, 1965), a sophisticated model that cares the order of words such as hierarchical Pitman-Yor processes (Wood et al, 2011), and smoothing/backoff methods to handle the sparseness problem.…”

“…The frequencies of high-order k-grams tend to be lower than in reality. We might need to place a hierarchical assumption on the a power law, as in done in hierarchical Pitman-Yor processes (Wood et al, 2011). Fig.…”

Temporal Sequence Modeling for Video Event Detection

“…To demonstrate the efficacy of exploiting long-range temporal dependencies in modeling, we compared the performance of our proposed approach with the n-gram temporal model [22]. Fig 6 clearly illustrates the benefit of modeling long-range temporal contexts in the data.…”

“…On the other hand, if k = i − 1, it puts the entire history of the sequence into consideration. Addressing such long contexts is a valid concern in previous work [22]. However, a recently developed probabilistic model [21] broke through this limitation by exploring an infinite length of context in a discrete data sequence.…”

Latent IBP Compound Dirichlet Allocation

“…in [19] and for more general sequence modelling in [20]. It was also shown to have a remarkable connection to interpolated Kneser-Ney, which is currently still one of the most effective language models since it was first proposed more than a decade ago [21], [22].…”