An Iteratively Reweighted Norm Algorithm for Minimization of Total Variation Functionals

We study theoretical runtime guarantees for a class of optimization problems that occur in a wide variety of inference problems. These problems are motivated by the LASSO framework and have applications in machine learning and computer vision.Our work shows a close connection between these problems and core questions in algorithmic graph theory. While this connection demonstrates the difficulties of obtaining runtime guarantees, it also suggests an approach of using techniques originally developed for graph algorithms.We show that most of these problems can be formulated as a grouped least squares problem, and give efficient algorithms for this formulation. Our algorithms rely on routines for solving quadratic minimization problems, which in turn are equivalent to solving linear systems. Some preliminary experimental work on image processing tasks are also presented.

show abstract

“…This method does a much more aggressive adjustment each iteration and to give good performances in practice [43].…”

Section: Iterative Reweighted Least Squaresmentioning

confidence: 99%

Runtime guarantees for regression problems

Chin

Mądry

Miller

et al. 2013

Proceedings of the 4th Conference on Innovations in Theoretical Computer Science

View full text Add to dashboard Cite

show abstract

“…To solve the convex nonsmooth optimization problem (1), we mostly find in the literature iterative fixed-point methods [11]. Until not so long ago, such methods applied to TV regularization had rather high computational complexity [12]- [16], but the growing interest for related ℓ 1 -norm problems in compressed sensing or sparse recovery [17], [18] has yielded advances in the field. Recent iterative methods based on operator splitting, which exploit both the primal and dual formulations of the problems and use variable stepsize strategies or Nesterov-style accelerations, are quite efficient when applied to TV-based problems [19]- [21].…”

Section: Introductionmentioning

confidence: 99%

A Direct Algorithm for 1-D Total Variation Denoising

Condat

2013

IEEE Signal Process. Lett.

297

280

View full text Add to dashboard Cite

Abstract-A very fast noniterative algorithm is proposed for denoising or smoothing one-dimensional discrete signals, by solving the total variation regularized least-squares problem or the related fused lasso problem. A C code implementation is available on the web page of the author.Index Terms-Total variation, denoising, nonlinear smoothing, fused lasso, regularized least-squares, nonparametric regression, convex nonsmooth optimization, taut string I. INTRODUCTIONThe problem of smoothing a signal, to remove or at least attenuate the noise it contains, has numerous applications in communications, control, machine learning, and many other fields of engineering and science [1]. In this paper, we focus on the numerical implementation of total variation (TV) denoising for one-dimensional (1D) discrete signals; that is, we are given a (noisy) signalN of size N ≥ 1, and we want to efficiently compute the denoised signal x ⋆ ∈ R N , defined implicitly as the solution to the minimization problemfor some regularization parameter λ ≥ 0 (whose choice is a difficult problem by itself [2]). We recall that, as the functional to minimize is strongly convex, the solution x ⋆ to the problem exists and is unique, whatever the data y. The TV denoising problem has received large attention in the communities of signal and image processing, inverse problems, sparse sampling, statistical regression analysis, optimization theory, among others. It is not the purpose of this paper to review the properties of the nonlinear TV denoising filter, since numerous papers can be found on this vast topic; see, e.g., [3]-[6] for various insights. A more general problem, which encompasses TV denoising as a particular case, is the fused lasso signal approximator, introduced in [7], which yields a solution that has sparsity in both the coefficients and their successive differences. It consists in solving the problem Total variation denoising can be interpreted as pulling the discrete primitive r of the signal y taut in a tube around it. The taut string (blue polyline) interpolates the sequence s ⋆ solution to (4) (blue dots), which after discrete differentiation yields the denoised sequence x ⋆ solution to (1). The proposed algorithm is not based on this interpretation.denoising, since the solution z ⋆ can be obtained by simple soft-thresholding from the solution x ⋆ of (1):It is straightforward to add soft-thresholding steps to the proposed algorithm, for essentially the same computation time. So, for simplicity of the exposition, we focus on the TV denoising problem (1) direct, noniterative, way, possibly in-place. It is appropriate for real-time processing of an incoming stream of data, as it locates the jumps in x ⋆ one after the other by forward scans, almost online. The possibility of such an algorithm sheds light on the relatively local nature of the TV denoising filter [25].After this work was completed, the author found that there already exists a direct, linear time, method for 1D TV denoising, called the taut string algorithm [26], see al...

show abstract

“…Here the iterative technique using the MS stabilizing operator which is frequently used for geophysical data inversion is considered. Its connection with the iteratively regularized norm algorithms, discussed in [32], has apparently not been previously noted in the literature but does demonstrate the convergence of the iteration based on the updating MS stabilizing operator L:…”

Section: Problem Tomomentioning

confidence: 89%

Regularization parameter estimation for underdetermined problems by the χ ² principle with application to 2D focusing gravity inversion

Vatankhah¹,

Renaut²,

Ardestani³

2014

Inverse Problems

View full text Add to dashboard Cite

Abstract. The χ 2 -principle generalizes the Morozov discrepancy principle to the augmented residual of the Tikhonov regularized least squares problem. For weighting of the data fidelity by a known Gaussian noise distribution on the measured data and, when the stabilizing, or regularization, term is considered to be weighted by unknown inverse covariance information on the model parameters, the minimum of the Tikhonov functional becomes a random variable that follows a χ 2 -distribution with m + p − n degrees of freedom for the model matrix G of size m × n and regularizer L of size p × n.Here it is proved that the result holds for the underdetermined case, m < n provided that m + p ≥ n and that the null spaces of the operators do not intersect. A Newton root-finding algorithm is used to find the regularization parameter α which yields the optimal inverse covariance weighting in the case of a white noise assumption on the mapped model data. It is implemented for small-scale problems using the generalized singular value decomposition, or singular value decomposition when L = I. Numerical results verify the algorithm for the case of regularizers approximating zero to second order derivative approximations, contrasted with the methods of generalized cross validation and unbiased predictive risk estimation. The inversion of underdetermined 2D focusing gravity data produces models with non-smooth properties, for which typical implementations in this field use the iterative minimum support stabilizer and both regularizer and regularizing parameter are updated each iteration. For a simulated data set with noise, the regularization parameter estimation methods for underdetermined data sets are used in this iterative framework, also contrasted with the L-curve and the Morozov Discrepancy principle. These experiments demonstrate the efficiency and robustness of the χ 2 -principle in this context, moreover showing that the L-curve and Morozov Discrepancy Principle are outperformed in general by the three other techniques. Furthermore, the minimum support stabilizer is of general use for the χ 2 -principle when implemented without the desirable knowledge of a mean value of the model.

show abstract

An Iteratively Reweighted Norm Algorithm for Minimization of Total Variation Functionals

Cited by 103 publications

References 17 publications

Runtime guarantees for regression problems

Runtime guarantees for regression problems

A Direct Algorithm for 1-D Total Variation Denoising

Regularization parameter estimation for underdetermined problems by the χ ² principle with application to 2D focusing gravity inversion

Contact Info

Product

Resources

About

An Iteratively Reweighted Norm Algorithm for Minimization of Total Variation Functionals

Cited by 103 publications

References 17 publications

Runtime guarantees for regression problems

Runtime guarantees for regression problems

A Direct Algorithm for 1-D Total Variation Denoising

Regularization parameter estimation for underdetermined problems by the χ 2 principle with application to 2D focusing gravity inversion

Contact Info

Product

Resources

About

Regularization parameter estimation for underdetermined problems by the χ ² principle with application to 2D focusing gravity inversion