A comparison of algorithms for maximum entropy parameter estimation

Abstract

Conditional maximum entropy (ME) models provide a general purpose machine learning technique which has been successfully applied to fields as diverse as computer vision and econometrics, and which is used for a wide variety of classification problems in natural language processing. However, the flexibility of ME models is not without cost. While parameter estimation for ME models is conceptually straightforward, in practice ME models for typical natural language tasks are very large, and may well contain many thousands of free parameters. In this paper, we consider a number of algorithms for estimating the parameters of ME models, including iterative scaling, gradient ascent, conjugate gradient, and variable metric methods. Surprisingly, the standardly used iterative scaling algorithms perform quite poorly in comparison to the others, and for all of the test problems, a limitedmemory variable metric algorithm outperformed the other choices.

Unsupervised labelling of sequential data for location identification in indoor environments

“…Thus, we iterate between belief propagation, analytical Maximum Likelihood updates and Limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) in an expectation maximization routine, helping scalability and handling computational restrictions. We note that L-BFGS is, to date, the preferred algorithm for fitting maximum-entropy models and CRFs, and has largely surpassed generalized iterative scaling algorithms for use with MEMMs (Malouf (2002)). …”

Self-Trained Bilexical Preferences to Improve Disambiguation Accuracy

“…Formal details of the disambiguation model are presented in [25]. For training the maximum entropy models, we use an implementation by [14].…”

Conditional Random Fields for Predicting and Analyzing Histone Occupancy, Acetylation and Methylation Areas in DNA Sequences

“…The traditional method for training in CRFs is iterative scaling algorithms [6,21]. Sine those methods are very slow for classification [20], therefore we use quasi-Newton methods, such as L-BFGS [8], which are significantly more efficient [10,20].…”