Damped Anderson Acceleration With Restarts and Monotonicity Control for Accelerating EM and EM-like Algorithms

Abstract

The expectation-maximization (EM) algorithm is a well-known iterative method for computing maximum likelihood estimates in a variety of statistical problems. Despite its numerous advantages, a main drawback of the EM algorithm is its frequently observed slow convergence which often hinders the application of EM algorithms in high-dimensional problems or in other complex settings. To address the need for more rapidly convergent EM algorithms, we describe a new class of acceleration schemes that build on the Anderson acceleration technique for speeding fixed-point iterations. Our approach is effective at greatly accelerating the convergence of EM algorithms and is automatically scalable to high dimensional settings. Through the introduction of periodic algorithm restarts and a damping factor, our acceleration scheme provides faster and more robust convergence when compared to un-modified Anderson acceleration, while also improving global convergence. Crucially, our method works as an "off-the-shelf" method in that it may be directly used to accelerate any EM algorithm without relying on the use of any model-specific features or insights. Through a series of simulation studies involving five representative problems, we show that our algorithm is substantially faster than the existing state-of-art acceleration schemes.

Alternating cyclic extrapolation methods for optimization algorithms

“…For QNAMM and DAAREM, the need to check the objective and their general computational burdens clearly played a major role. Henderson and Varadhan (2019)'s results suggest DAAREM should perform better for EM-like problems with more parameters.…”

“…For acceleration of general fixed-point iterations, the four problems selected are the expectation maximization (EM) for Poisson admixture, alternating least squares (ALS) for canonical tensor decomposition, the power method for computing dominant eigenvalues and the method of alternating projections (von Neumann (1950), Halperin (1962)) applied to regression with high-dimensional fixed effects. The performances of ACX is compared to competitive general purpose acceleration algorithms: the quasi-Newton acceleration of Zhou et al (2011), the objective acceleration approach of Riseth (2019) and the Anderson Acceleration version of Henderson and Varadhan (2019). For the high-dimensional fixed-effect regression, ACX is compared with equivalent packages in various programming languages.…”

A Variational EM Acceleration for Efficient Clustering at Very Large Scales

“…iterations : The number of iterations can be reduced by advanced parameter initializations (seeding) [38], and seeding methods have recently been made very efficient [39], [40], [41]. Furthermore, iteration numbers of EMlike algorithms have been reduced using restarts and monotonicity control [42] or steplength optimization [43]. In this work, we aim at optimizing GMMs with diagonal covariances on very large-scale datasets, where large refers to large N and C (as well as potentially large D).…”

An Efficient Approach for Parameters Learning of Bayesian Network with Multiple Latent Variables Using Neural Networks and P-EM

“…Quasi Newton [11] was also used to make EM algorithm more efficient by adopting Quasi Newton function to generate an approximation instead of an accurate estimation. The Damped Anderson acceleration method [10] was further proposed by accelerating the iterations. Constraints were used to reduce the search space during the learning of parameters and structure [7,8].…”