A Two-Step Fixed-Point Proximity Algorithm for a Class of Non-differentiable Optimization Models in Machine Learning

Zheng, Lianxi; Song, Guohui; Xu, Yuesheng

doi:10.1007/s10915-019-01045-7

Cited by 6 publications

(9 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is desirable to develop alternative representations of solutions of the minimum norm interpolation problem convenient for algorithmic development. Motivated from the success of the fixed-point approach used in machine learning [2,44,45,46,56], image processing [11,41,42,47,51], medical imaging [40,43,85] and solutions of inverse problems [27,37], we will develop representations of a solution of the minimum norm interpolation problem or the regularization problem in a Banach space, as fixed-points of nonlinear maps defined by proximity operators of functions involved in the problem. The fixed-point formulation well fits for designing iterative algorithms.…”

Section: Solutions Of Minimum Norm Interpolation In a Banach Space Wi...mentioning

confidence: 99%

“…for y := [y j : j ∈ N m ] ∈ {−1, 1} m and the regularizer as (7.4), the regularization problem (7.2) describes the support vector machine classification. Moreover, ℓ 1 support vector machine regression and classification [45,46,59,86] are formulated as (7.2) with the loss function (7.5) and (7.6), respectively, and the regularizer…”

Section: Regularization Problemmentioning

confidence: 99%

“…It is desirable to develop solution representations of the minimum norm interpolation problem and the regularization problem convenient for algorithmic development. Inspired by the suc-cess of the fixed-point approach used in solving several types of finite dimensional problems such as machine learning [2,44,45,46,56], image processing [11,41,42,47,51], medical imaging [40,43,85] and solutions of semi-discrete inverse problems [27,37], we develop solution representations for the minimum norm interpolation problem and the regularization problem by using a fixed-point formulation via the proximity operator of the functions appearing in the objective function or constraints. This formulation has great potential for convenience of designing iterative algorithms for solving these problems.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Representer Theorems in Banach Spaces: Minimum Norm Interpolation, Regularized Learning and Semi-Discrete Inverse Problems

Wang,

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Constructing or learning a function from a finite number of sampled data points (measurements) is a fundamental problem in science and engineering. This is often formulated as a minimum norm interpolation problem, regularized learning problem or, in general, a semidiscrete inverse problem, in certain functional spaces. The choice of an appropriate space is crucial for solutions of these problems. When they are considered in an infinite dimensional Hilbert space, which gives a linear method for their solutions, the remarkable representer theorem reduces their solutions to finding a finite number of coefficients of elements in the space. Motivated by sparse representations of the reconstructed functions such as compressed sensing and sparse learning, much of the recent research interest has been directed to considering these problems in certain Banach spaces in order to obtain their sparse solutions, which is a feasible approach to overcome challenges coming from the big data nature of most practical applications. It is the goal of this paper to provide a systematic study of the representer theorems for these problems in Banach spaces. There are a few existing results for these problems in a Banach space, with all of them regarding implicit representer theorems. We aim at obtaining explicit representer theorems based on which convenient solution methods will then be developed. For the minimum norm interpolation, the explicit representer theorems enable us to express the infimum in terms of the norm of the linear combination of the interpolation functionals. For the purpose of developing efficient computational algorithms, we establish the fixed-point equation formulation of solutions of these problems. We reveal that unlike in a Hilbert space, in general, solutions of these problems in a Banach space may not be able to be reduced to truly finite dimensional problems (with certain infinite dimensional components hidden). We demonstrate how this obstacle can be removed, reducing the original problem to a truly finite dimensional one, in the special case when the Banach space is ℓ 1 (N).

show abstract

Section: Solutions Of Minimum Norm Interpolation In a Banach Space Wi...mentioning

confidence: 99%

Section: Regularization Problemmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Representer Theorems in Banach Spaces: Minimum Norm Interpolation, Regularized Learning and Semi-Discrete Inverse Problems

Wang,

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…We consider in this paper the convergence rate analysis of fixed-point algorithms. Fixed-point type algorithms have been popular in solving nondifferentiable convex or nonconvex optimization problems such as image processing [16,25,30,32,33,41], medical imaging [24,29,38,47], machine learning [14,27,28,36], and compressed sensing [21,48]. Existing fixed-point type algorithms for optimization including the gradient descent algorithm [8,39], the proximal point algorithm [37], the proximal gradient algorithm [7,35], the forward-backward splitting algorithm [15,45] and the fixed-point proximity algorithm [25,29,32,33].…”

mentioning

confidence: 99%

“…This type of optimization models is raised from machine learning (e.g. 1 -SVM, LASSO regression) [28], compressed sensing [21] and image processing [6,20].…”

mentioning

confidence: 99%

Convergence Rate Analysis for Fixed-Point Iterations of Generalized Averaged Nonexpansive Operators

Lin¹,

Xu²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

We estimate convergence rates for fixed-point iterations of a class of nonlinear operators which are partially motivated from solving convex optimization problems. We introduce the notion of the generalized averaged nonexpansive (GAN) operator with a positive exponent, and provide a convergence rate analysis of the fixed-point iteration of the GAN operator. The proposed generalized averaged nonexpansiveness is weaker than the averaged nonexpansiveness while stronger than nonexpansiveness. We show that the fixed-point iteration of a GAN operator with a positive exponent converges to its fixed-point and estimate the local convergence rate (the convergence rate in terms of the distance between consecutive iterates) according to the range of the exponent. We prove that the fixed-point iteration of a GAN operator with a positive exponent strictly smaller than 1 can achieve an exponential global convergence rate (the convergence rate in terms of the distance between an iterate and the solution). Furthermore, we establish the global convergence rate of the fixed-point iteration of a GAN operator, depending on both the exponent of generalized averaged nonexpansiveness and the exponent of the Hölder regularity, if the GAN operator is also Hölder regular. We then apply the established theory to three types of convex optimization problems that appear often in data science to design fixed-point iterative algorithms for solving these optimization problems and to analyze their convergence properties.

show abstract

Inexact Fixed-Point Proximity Algorithm for the $$\ell _0$$ Sparse Regularization Problem

Fang,

Xu,

Yan

2024

J Sci Comput

View full text Add to dashboard Cite

We study inexact fixed-point proximity algorithms for solving a class of sparse regularization problems involving the $$\ell _0$$ ℓ 0 norm. Specifically, the $$\ell _0$$ ℓ 0 model has an objective function that is the sum of a convex fidelity term and a Moreau envelope of the $$\ell _0$$ ℓ 0 norm regularization term. Such an $$\ell _0$$ ℓ 0 model is non-convex. Existing exact algorithms for solving the problems require the availability of closed-form formulas for the proximity operator of convex functions involved in the objective function. When such formulas are not available, numerical computation of the proximity operator becomes inevitable. This leads to inexact iteration algorithms. We investigate in this paper how the numerical error for every step of the iteration should be controlled to ensure global convergence of the inexact algorithms. We establish a theoretical result that guarantees the sequence generated by the proposed inexact algorithm converges to a local minimizer of the optimization problem. We implement the proposed algorithms for three applications of practical importance in machine learning and image science, which include regression, classification, and image deblurring. The numerical results demonstrate the convergence of the proposed algorithm and confirm that local minimizers of the $$\ell _0$$ ℓ 0 models found by the proposed inexact algorithm outperform global minimizers of the corresponding $$\ell _1$$ ℓ 1 models, in terms of approximation accuracy and sparsity of the solutions.

show abstract

A Two-Step Fixed-Point Proximity Algorithm for a Class of Non-differentiable Optimization Models in Machine Learning

Cited by 6 publications

References 20 publications

Representer Theorems in Banach Spaces: Minimum Norm Interpolation, Regularized Learning and Semi-Discrete Inverse Problems

Representer Theorems in Banach Spaces: Minimum Norm Interpolation, Regularized Learning and Semi-Discrete Inverse Problems

Convergence Rate Analysis for Fixed-Point Iterations of Generalized Averaged Nonexpansive Operators

Inexact Fixed-Point Proximity Algorithm for the $$\ell _0$$ Sparse Regularization Problem

Contact Info

Product

Resources

About