A Generalization of the Chambolle-Pock Algorithm to Banach Spaces with Applications to Inverse Problems

Hohage, Thorsten; Homann, Carolin

doi:10.48550/arxiv.1412.0126

Cited by 6 publications

(8 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since then, many variants and extensions of the PDHG algorithm have been proposed; see [12,14,15] for further details and references. A partial list of these variants include: overrelaxed [18,38], inertial [52], operator, forward-backward, and proximal-gradient splitting [7,17,21,22,71], multistep [16], stochastic [56,15,29,58,70,72,73], and nonlinear [14,43] variants, including the mirror descent method [54]. Here, we focus on nonlinear variants of the PDHG algorithm.…”

Section: Related Workmentioning

confidence: 99%

“…The extension of the PDHG algorithm to the nonlinear setting was first done by Hohage and Homann [43] to solve non-smooth convex optimization problems posed on Banach spaces. A nonlinear PDHG algorithm for solving such problems using nonlinear proximity operators based on Bregman divergences was later proposed by Chambolle and Pock [14].…”

Section: Related Workmentioning

confidence: 99%

“…Due to the strong convexity of the primal problem (42) and strong concavity of the dual problem (43), the convex-concave saddle-point problem (44) has a unique saddle point (x s , y * s ) ∈ R n × R m . Here x s and y * s are the unique solutions to the primal and dual problems above.…”

Section: Kernel Ridge Regressionmentioning

confidence: 99%

See 2 more Smart Citations

Accelerated nonlinear primal-dual hybrid gradient methods with applications to supervised machine learning

Darbon¹,

Langlois²

2021

Preprint

View full text Add to dashboard Cite

The primal-dual hybrid gradient (PDHG) algorithm is a first-order method that splits convex optimization problems with saddle-point structure into smaller subproblems. Those subproblems, unlike those obtained from most other splitting methods, can generally be solved efficiently because they involve simple operations such as matrix-vector multiplications or proximal mappings that are easy to evaluate. In order to work fast, however, the PDHG algorithm requires stepsize parameters fine-tuned for the problem at hand. Unfortunately, the stepsize parameters must often be estimated from quantities that are prohibitively expensive to compute for large-scale optimization problems, such as those in machine learning. In this paper, we introduce accelerated nonlinear variants of the PDHG algorithm that can achieve, for a broad class of optimization problems relevant to machine learning, an optimal rate of convergence with stepsize parameters that are simple to compute. We prove rigorous convergence results, including for problems posed on infinite-dimensional reflexive Banach spaces. We also provide practical implementations of accelerated nonlinear PDHG algorithms for solving several regression tasks in machine learning, including support vector machines without offset, kernel ridge regression, elastic net regularized linear regression, and the least absolute shrinkage selection operator.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Accelerated nonlinear primal-dual hybrid gradient methods with applications to supervised machine learning

Darbon¹,

Langlois²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…This paper proposes a novel optimization algorithm that addresses the shortcomings of state-ofthe-art algorithms used for variable selection. Our proposed algorithm is an accelerated nonlinear variant of the classic primal-dual hybrid gradient (PDHG) algorithm, a first-order optimization method initially developed to solve imaging problems [23,54,78,12,35,13]. Our proposed accelerated nonlinear PDHG algorithm, which is based on the work the authors recently provided in [16], uses the Kullback-Leibler divergence to efficiently fit a logistic regression model regularized by a convex combination of ℓ 1 and ℓ 2 2 penalties.…”

Section: Introductionmentioning

confidence: 99%

Efficient and robust high-dimensional sparse logistic regression via nonlinear primal-dual hybrid gradient algorithms

Darbon¹,

Langlois²

2021

Preprint

View full text Add to dashboard Cite

Logistic regression is a widely used statistical model to describe the relationship between a binary response variable and predictor variables in data sets. It is often used in machine learning to identify important predictor variables. This task, variable selection, typically amounts to fitting a logistic regression model regularized by a convex combination of ℓ 1 and ℓ 2 2 penalties. Since modern big data sets can contain hundreds of thousands to billions of predictor variables, variable selection methods depend on efficient and robust optimization algorithms to perform well. State-of-the-art algorithms for variable selection, however, were not traditionally designed to handle big data sets; they either scale poorly in size or are prone to produce unreliable numerical results. It therefore remains challenging to perform variable selection on big data sets without access to adequate and costly computational resources. In this paper, we propose a nonlinear primal-dual algorithm that addresses these shortcomings. Specifically, we propose an iterative algorithm that provably computes a solution to a logistic regression problem regularized by an elastic net penalty in O(T (m, n) log(1/ǫ)) operations, where ǫ ∈ (0, 1) denotes the tolerance and T (m, n) denotes the number of arithmetic operations required to perform matrix-vector multiplication on a data set with m samples each comprising n features. This result improves on the known complexity bound of O(min(m 2 n, mn 2 ) log(1/ǫ)) for first-order optimization methods such as the classic primal-dual hybrid gradient or forward-backward splitting methods. Significance statementLogistic regression is a widely used statistical model to describe the relationship between a binary response variable and predictor variables in data sets. With the trends in big data, logistic regression is now commonly applied to data sets whose predictor variables range from hundreds of thousands to billions. State-of-the-art algorithms for fitting logistic regression models, however, were not traditionally designed to handle big data sets; they either scale poorly in size or are prone to produce unreliable numerical results. This paper proposes a nonlinear primal-dual algorithm that provably computes a solution to a logistic regression problem regularized by an elastic net penalty in O(T (m, n) log(1/ǫ)) operations, where ǫ ∈ (0, 1) denotes the tolerance and T (m, n) denotes the number of arithmetic operations required to perform matrix-vector multiplication on a data set with m samples each comprising n features. This result improves on the known complexity bound of O(min(m 2 n, mn 2 ) log(1/ǫ)) for first-order optimization methods such as the classic primal-dual hybrid gradient or forward-backward splitting methods.

show abstract

“…We generally have to consider discretisations, since many interesting infinite-dimensional problems necessitate Banach spaces. Using Bregman distances, it would be possible to generalise our work form Hilbert spaces to Banach spaces, as was done in [22] for the original method of [11]. This is however outside the scope of the present work.…”

mentioning

confidence: 99%

Acceleration of the PDHGM on Partially Strongly Convex Functions

Valkonen

Pock

2016

J Math Imaging Vis

View full text Add to dashboard Cite

We propose several variants of the primal-dual method due to Chambolle and Pock. Without requiring full strong convexity of the objective functions, our methods are accelerated on subspaces with strong convexity. This yields mixed rates, O(1/N 2 ) with respect to initialisation and O(1/N ) with respect to the dual sequence, and the residual part of the primal sequence. We demonstrate the efficacy of the proposed methods on image processing problems lacking strong convexity, such as total generalised variation denoising and total variation deblurring

show abstract

A Generalization of the Chambolle-Pock Algorithm to Banach Spaces with Applications to Inverse Problems

Cited by 6 publications

References 24 publications

Accelerated nonlinear primal-dual hybrid gradient methods with applications to supervised machine learning

Accelerated nonlinear primal-dual hybrid gradient methods with applications to supervised machine learning

Efficient and robust high-dimensional sparse logistic regression via nonlinear primal-dual hybrid gradient algorithms

Acceleration of the PDHGM on Partially Strongly Convex Functions

Contact Info

Product

Resources

About