Separable nonlinear least squares: the variable projection method and its applications

Abstract

In this paper we review 30 years of developments and applications of the variable projection method for solving separable nonlinear least-squares problems. These are problems for which the model function is a linear combination of nonlinear functions. Taking advantage of this special structure, the method of variable projections eliminates the linear variables obtaining a somewhat more complicated function that involves only the nonlinear parameters. This procedure not only reduces the dimension of the parameter space but also results in a better-conditioned problem. The same optimization method applied to the original and reduced problems will always converge faster for the latter. We present first a historical account of the basic theoretical work and its various computer implementations, and then report on a variety of applications from electrical engineering, medical and biological imaging, chemistry, robotics, vision, and environmental sciences. An extensive bibliography is included. The method is particularly well suited for solving real and complex exponential model fitting problems, which are pervasive in their applications and are notoriously hard to solve.

Variable selection for sparse high-dimensional nonlinear regression models by combining nonnegative garrote and sure independence screening

“…The above iterative algorithm can be viewed as a solution of a separable nonlinear least squares problem (Ruhe and Wedin, 1980; Golub and Pereyra, 2003). The parameters are separated into two sets, where { β j } are linear parameters and { α j } are nonlinear parameters.…”

“…It has been shown that eliminating one set of parameters can result in faster convergence of the optimization problem. We refer to Ruhe and Wedin (1980) and Golub and Pereyra (2003) and references therein for detailed descriptions of separable nonlinear least squares problems and the convergence properties of related algorithms.…”

Optimal Registration Of Aliased Images Using Variable Projection With Applications To Super-Resolution

“…The form of Equation (5) is known as the variable projection functional [25] as the operator B(B T B) 21 B T is a projection operator. This form of the cost function has several advantages over the original cost function of Equation (3).…”

“…While this second approach is more principled than the two-stage approach, we show that the use of cyclic coordinate descent is not the most efficient search strategy. It has been shown [25], that optimization using the coordinate descent approach is numerically less well conditioned and likely to become stuck in local minima.…”

“…Our approach is a multiframe registration solution based on the principle of variable projections [25]. In Section 2, we describe how to avoid the shortcomings of the cyclic coordinate descent using this variable projection formulation and present an efficient implementation for the case of translational motion.…”