Optimization Transfer Using Surrogate Objective Functions

Lange, Kenneth; Hunter, David R.; Yang, Ilsoon

doi:10.1080/10618600.2000.10474858

Cited by 438 publications

(442 citation statements)

References 24 publications

Supporting

Mentioning

442

Contrasting

Order By: Relevance

“…For this reason, we borrow a technique of optimization transfer (see, e.g., [12,29]) and construct surrogate functionals that effectively remove the term K * K f . We first pick a constant C so that K * K < C, and then we define the functional ( f ; a) = C f − a 2 − K f − K a 2 , which depends on an auxiliary element a of H. Because CI − K * K is a strictly positive operator, ( f ; a) is strictly convex in f for any choice of a.…”

Section: The Iterative Algorithm: a Derivation From Surrogate Functiomentioning

confidence: 99%

An iterative thresholding algorithm for linear inverse problems with a sparsity constraint

Daubechies

Defrise

Mol

2004

Comm Pure Appl Math

4,081

3,559

View full text Add to dashboard Cite

We consider linear inverse problems where the solution is assumed to have a sparse expansion on an arbitrary preassigned orthonormal basis. We prove that replacing the usual quadratic regularizing penalties by weighted p -penalties on the coefficients of such expansions, with 1 ≤ p ≤ 2, still regularizes the problem. Use of such p -penalized problems with p < 2 is often advocated when one expects the underlying ideal noiseless solution to have a sparse expansion with respect to the basis under consideration. To compute the corresponding regularized solutions, we analyze an iterative algorithm that amounts to a Landweber iteration with thresholding (or nonlinear shrinkage) applied at each iteration step. We prove that this algorithm converges in norm.

show abstract

Section: The Iterative Algorithm: a Derivation From Surrogate Functiomentioning

confidence: 99%

An iterative thresholding algorithm for linear inverse problems with a sparsity constraint

Daubechies

Defrise

Mol

2004

Comm Pure Appl Math

4,081

3,559

View full text Add to dashboard Cite

show abstract

“…Iteration then results in convergence to the unique maximum. In [29] the MM method was applied to the least absolute deviation regression problem, leading to a viable surrogate for the absolute value penalty. By combining these ideas [8] provides an iterative algorithm for the maximization of the log-likelihood function with sparsity inducing penalty in the multinomial case -this is the method used here in its binomial form.…”

Section: Methods and Theorymentioning

confidence: 99%

Automatic covariate selection in logistic models for chest pain diagnosis: A new approach

Harrison

Kennedy

2008

Computer Methods and Programs in Biomedicine

View full text Add to dashboard Cite

A newly established method for optimizing logistic models via a minorizationmajorization procedure is applied to the problem of diagnosing acute coronary syndromes (ACS). The method provides a principled approach to the selection of covariates which would otherwise require the use of a suboptimal method owing to the size of the covariate set. A strategy for building models is proposed and two models optimized for performance and for simplicity are derived via ten-fold crossvalidation. These models confirm that a relatively small set of covariates including clinical and electrocardiographic features can be used successfully in this task.The performance of the models is comparable with previously published models using less principled selection methods. The models prove to be portable when tested on data gathered from three other sites. Whilst diagnostic accuracy and calibration diminishes slightly for these new settings, it remains satisfactory overall.The prospect of building predictive models that are as simple as possible for a required level of performance is valuable if data-driven decision aids are to gain wide acceptance in the clinical situation owing to the need to minimize the time taken to gather and enter data at the bedside.

show abstract

“…Minorization-maximization algorithms (MM; Lange et al, 2000), also known as bound optimization algorithms (see, e.g., Salakhutdinov and Roweis, 2003), are extensions of EM that do not require a missing data framework. To maximize l(θ) (e.g., the log-likelihood), we first find a function Q(θ|θ) such that l(θ) ≥ Q(θ|θ) for all θ andθ, with equality when θ =θ.…”

Section: Extensionsmentioning

confidence: 99%

“…Theoretical properties, such as monotonicity and convergence rate, are also analyzed in Section 3. We also discuss how it can be applied to more general algorithms such as the minorization-maximization (or majorization-minimization) algorithm (Lange et al, 2000). Section 4 applies monotonic overrelaxation to several examples including least absolute deviations regression, Poisson inverse problems, and finite mixtures.…”

Section: Introductionmentioning

confidence: 99%

Monotonically Overrelaxed EM Algorithms

Yu¹

2012

Journal of Computational and Graphical Statistics

View full text Add to dashboard Cite

We explore the idea of overrelaxation for accelerating the expectation-maximization (EM) algorithm, focusing on preserving its simplicity and monotonic convergence properties. It is shown that in many cases a trivial modification in the M-step results in an algorithm that maintains monotonic increase in the log-likelihood, but can have an appreciably faster convergence rate, especially when EM is very slow. The method is applicable to more general fixed point algorithms. Its simplicity and effectiveness are illustrated with several statistical problems, including probit regression, least absolute deviations regression, Poisson inverse problems, and finite mixtures. This paper has supplemental materials online.

show abstract

Optimization Transfer Using Surrogate Objective Functions

Cited by 438 publications

References 24 publications

An iterative thresholding algorithm for linear inverse problems with a sparsity constraint

An iterative thresholding algorithm for linear inverse problems with a sparsity constraint

Automatic covariate selection in logistic models for chest pain diagnosis: A new approach

Monotonically Overrelaxed EM Algorithms

Contact Info

Product

Resources

About