Inducing probabilistic grammars by Bayesian model merging

Abstract

We describe a framework for inducing probabilistic grammars from corpora of positive samples. First, samples are incorporated by adding ad-hoc rules to a working grammar; subsequently, elements of the model (such as states or nonterminals) are merged to achieve generalization and a more compact representation. The choice of what to merge and when to stop is governed by the Bayesian posterior probability of the grammar given the data, which formalizes a trade-off between a close fit to the data and a default preference for simpler models ('Occam's Razor'). The general scheme is illustrated using three types of probabilistic grammars: Hidden Markov models, class-based ¢ -grams, and stochastic context-free grammars.

Comparing Hidden Markov Models and Long Short Term Memory Neural Networks for Learning Action Representations

“…For the HMM parameter estimation, we apply an incremental learning scheme utilizing the best first model merging framework [8,9]. Model merging is inspired by the observation that, when faced with new situations, humans and animals alike drive their learning process by first storing individual examples (memory based learning) when few data points are available and gradually switching to a parametric learning scheme to allow for better generalization as more and more data becomes available [10].…”

Computational Models of Language Acquisition

“…Stolcke and Omohundro [58] propose a technique called Bayesian Model Merging (BMM): first, strings that are observed in the data are incorporated by adding ad-hoc rules to form an initial grammar; then, the grammar is made more concise by merging some of the rules. Stolcke and Omohundro [58] discuss two incarnations of their technique, one in which the models are probabilistic context-free grammars (PCFGs), and another in which they are hidden Markov models (HMMs).…”

“…Stolcke and Omohundro [58] discuss two incarnations of their technique, one in which the models are probabilistic context-free grammars (PCFGs), and another in which they are hidden Markov models (HMMs). In the former, rules are merged by identifying non-terminal symbols A and B if the rule A → B is in the grammar; this leads to (over-) generalizations, and renders the grammar more compact.…”

“…In both cases, the prior probabilities are optimized by minimizing their description length. Stolcke and Omohundro [58] discuss the application of BMM to natural language learning, but do not provide quantitative evaluation results.…”

“…Notably, Borensztajn and Zuidema [16] extend the BMM model of Stolcke and Omohundro [58], but they assume that the input is already bracketed. Their algorithm then proceeds by merging nonterminal labels to maximize a Bayesian objective function.…”