On Prediction Using Variable Order Markov Models

Abstract

This paper is concerned with algorithms for prediction of discrete sequences over a finite alphabet, using variable order Markov models. The class of such algorithms is large and in principle includes any lossless compression algorithm. We focus on six prominent prediction algorithms, including Context Tree Weighting (CTW), Prediction by Partial Match (PPM) and Probabilistic Suffix Trees (PSTs). We discuss the properties of these algorithms and compare their performance using real life sequences from three domains: proteins, English text and music pieces. The comparison is made with respect to prediction quality as measured by the average log-loss. We also compare classification algorithms based on these predictors with respect to a number of large protein classification tasks. Our results indicate that a "decomposed" CTW (a variant of the CTW algorithm) and PPM outperform all other algorithms in sequence prediction tasks. Somewhat surprisingly, a different algorithm, which is a modification of the Lempel-Ziv compression algorithm, significantly outperforms all algorithms on the protein classification problems.

Measuring the Efficiency of the Intraday Forex Market with a Universal Data Compression Algorithm

“…The crucial essence in compression is estimating the conditional probability for the next outcome given the past observations, so those symbols (and sub-sequences) with high conditional probabilities are assigned short codes . long sequences, attain the entropy lower bound (Begleiter et al, 2004). Thus, constructing a data-compression model that minimizes the average log-loss score of a sequence is equivalent to constructing a prediction model that maximizes the likelihood of a sequence.…”

“…Therefore, in this study the terms "compression" and "prediction" are often considered as equivalent terms. Note that although other universal-compression algorithms can be used for prediction, the used VOM modela variation of Rissanen"s (1983) context tree -has been shown to attain the best asymptotic convergence rate for a given sequence (Ziv 2001, Begleiter et al, 2004.…”

“…The VOM model which we used here is a variant of the Prediction by Partial Match (PPM) tree, which was found in Begleiter et al (2004) to outperform other variants of the VOM model. Our version of the VOM model is different in its parameterization, growth, smoothing procedure and pruning stages from the previous versions of the model.…”

“…Our version of the VOM model is different in its parameterization, growth, smoothing procedure and pruning stages from the previous versions of the model. These differences might be significant, especially when applying the model to small datasets (Buhlmann and Wyner, 1999, Ziv, 2001, Begleiter et al, 2004). …”

“…We follow the explanations and style in Begleiter et al (2004) and Ben-Gal et al (2003) that contain further details on the model and its construction. and, hence, can be considered as special cases of the VOM models.…”