An Inexact Accelerated Proximal Gradient Method and a Dual Newton-CG Method for the Maximal Entropy Problem

Wang, Chengjing; Xu, Aiqiang

doi:10.1007/s10957-012-0150-2

Cited by 7 publications

(8 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, in the field of large scale optimization, there is a growing interest in inexact and approximate Newton-type methods for [7,11,1,40,39,13], which can benefit from fast subroutines for calculating approximate solutions of linear systems. In machine learning, applications arise for the problem of finding optimal configurations in Gaussian Markov Random Fields [32], in graph-based semi-supervised learning and other graph-Laplacian problems [2], least-squares SVMs, Gaussian processes and more.…”

mentioning

confidence: 99%

Randomized Iterative Methods for Linear Systems

Gower¹,

Richtárik²

2015

SIAM J. Matrix Anal. & Appl.

223

293

View full text Add to dashboard Cite

Abstract. We develop a novel, fundamental, and surprisingly simple randomized iterative method for solving consistent linear systems. Our method has six different but equivalent interpretations: sketch-and-project, constrain-and-approximate, random intersect, random linear solve, random update, and random fixed point. By varying its two parameters-a positive definite matrix (defining geometry), and a random matrix (sampled in an independent and identically distributed fashion in each iteration)-we recover a comprehensive array of well-known algorithms as special cases, including the randomized Kaczmarz method, randomized Newton method, randomized coordinate descent method, and random Gaussian pursuit. We naturally also obtain variants of all these methods using blocks and importance sampling. However, our method allows for a much wider selection of these two parameters, which leads to a number of new specific methods. We prove exponential convergence of the expected norm of the error in a single theorem, from which existing complexity results for known variants can be obtained. However, we also give an exact formula for the evolution of the expected iterates, which allows us to give lower bounds on the convergence rate.Key words. linear systems, stochastic methods, iterative methods, randomized Kaczmarz, randomized Newton, randomized coordinate descent, random pursuit, randomized fixed point AMS subject classifications. 15A06, 15B52, 65F10, 68W20, 65N75, 65Y20, 68Q25, 68W40, 90C20 DOI. 10.1137/15M10254871. Introduction. The need to solve linear systems of equations is ubiquitous in essentially all quantitative areas of human endeavor, including industry and science. Linear systems are a central problem in numerical linear algebra and play an important role in computer science, mathematical computing, optimization, signal processing, engineering, numerical analysis, computer vision, machine learning, and many other fields. For instance, in the field of large scale optimization, there is a growing interest in inexact and approximate Newton-type methods for [7,11,1,40,39,13], which can benefit from fast subroutines for calculating approximate solutions of linear systems. In machine learning, applications arise for the problem of finding optimal configurations in Gaussian Markov random fields [32], in graph-based semisupervised learning and other graph-Laplacian problems [2], in least-squares SVMs, in Gaussian processes, and in others.In a large scale setting, direct methods are generally not competitive when compared to iterative approaches. While classical iterative methods are deterministic, recent breakthroughs suggest that randomization can play a powerful role in the design and analysis of efficient algorithms [38,19,22,9,41,18,21,29] which are in many situations competitive or better than existing deterministic methods.

show abstract

mentioning

confidence: 99%

Randomized Iterative Methods for Linear Systems

Gower¹,

Richtárik²

2015

SIAM J. Matrix Anal. & Appl.

223

293

View full text Add to dashboard Cite

show abstract

“…It has been reported that the inexact scheme can achieve faster computational speeds and better local minima than can the exact scheme [22,24]. Moreover, the inexact scheme has been employed with APG for many problems in broad areas, such as quadratic semidefinite programming [25], maximal entropy [26], and tensor recovery [27]. Therefore, the inexact scheme and APG can naturally be extended to the NCP problem.…”

Section: Introductionmentioning

confidence: 99%

An inexact alternating proximal gradient algorithm for nonnegative CP tensor decomposition

Wang

Cong

2021

Sci. China Technol. Sci.

View full text Add to dashboard Cite

Nonnegative tensor decomposition has become increasingly important for multiway data analysis in recent years. The alternating proximal gradient (APG) is a popular optimization method for nonnegative tensor decomposition in the block coordinate descent framework. In this study, we propose an inexact version of the APG algorithm for nonnegative CANDECOMP/PARAFAC decomposition, wherein each factor matrix is updated by only finite inner iterations. We also propose a parameter warm-start method that can avoid the frequent parameter resetting of conventional APG methods and improve convergence performance. By experimental tests, we find that when the number of inner iterations is limited to around 10 to 20, the convergence speed is accelerated significantly without losing its low relative error. We evaluate our method on both synthetic and real-world tensors. The results demonstrate that the proposed inexact APG algorithm exhibits outstanding performance on both convergence speed and computational precision compared with existing popular algorithms.

show abstract

“…From a Machine Learning perspective, problem (1) arises from a wide range of applications such as Gaussian processes (Rasmussen and Williams, 2008), Least-Square Support Vector Machines (Ye and Xiong, 2007), graph-based Semi-Supervised Learning and Graph-Laplacian problems (Bengio et al, 2006), Gaussian Markov Random Fields (Rue and Held, 2005), etc. Approximate solutions of Linear Systems can be of practical benefit in inexact Newton schemes (Jiang et al, 2012;Wang and Xu, 2013;Gondzio, 2013) that are gaining lots of traction in the field of large-scale optimization. Throughout the paper, we assume that problem (1) is consistent (there exists x * such that Ax * = b), dimensions are large and m n. In a large-scale setting, solving problem (1) with direct methods are infeasible.…”

Section: Introductionmentioning

confidence: 99%

Stochastic Steepest Descent Methods for Linear Systems: Greedy Sampling & Momentum

Morshed,

Ahmad,

Noor-E-Alam

2020

Preprint

View full text Add to dashboard Cite

Recently proposed adaptive Sketch & Project (SP) methods connect several well-known projection methods such as Randomized Kaczmarz (RK), Randomized Block Kaczmarz (RBK), Motzkin Relaxation (MR), Randomized Coordinate Descent (RCD), Capped Coordinate Descent (CCD) etc. into one framework for solving linear systems. In this work, we first propose a Stochastic Steepest Descent (SSD) framework that connects SP methods with the well-known Steepest Descent (SD) method for solving positive-definite linear system of equations. We then introduce two greedy sampling strategies in the SSD framework that allow us to obtain algorithms such as Sampling Kaczmarz Motzkin (SKM), Sampling Block Kaczmarz (SBK), Sampling Coordinate Descent (SCD), etc. In doing so, we generalize the existing sampling rules into one framework and develop an efficient version of SP methods. Furthermore, we incorporated the Polyak momentum technique into the SSD method to accelerate the resulting algorithms. We provide global convergence results for both the SSD method and the momentum induced SSD method. Moreover, we prove O( 1 k ) convergence rate for the Cesaro average of iterates generated by both methods. By varying parameters in the SSD method, we obtain classical convergence results of the SD method as well as the SP methods as special cases. We design computational experiments to demonstrate the performance of the proposed greedy sampling methods as well as the momentum methods. The proposed greedy methods significantly outperform the existing methods for a wide variety of datasets such as random test instances as well as real-world datasets (LIBSVM, sparse datasets from matrix market collection). Finally, the momentum algorithms designed in this work accelerate the algorithmic performance of the SSD methods.

show abstract

An Inexact Accelerated Proximal Gradient Method and a Dual Newton-CG Method for the Maximal Entropy Problem

Cited by 7 publications

References 25 publications

Randomized Iterative Methods for Linear Systems

Randomized Iterative Methods for Linear Systems

An inexact alternating proximal gradient algorithm for nonnegative CP tensor decomposition

Stochastic Steepest Descent Methods for Linear Systems: Greedy Sampling & Momentum

Contact Info

Product

Resources

About