Stochastic Learning via Optimizing the Variational Inequalities

Qing, Tao; Gao, Qiankun; Chu, Dejun; Wu, Guangjian

doi:10.1109/tnnls.2013.2294741

Cited by 12 publications

(6 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Based on its equivalent reformulation, Nesterov's extrapolation can be employed in MD to achieve optimal individual convergence. Obviously, it is interesting that whether we can extend Nesterov's extrapolation to other optimization methods such as alternating direction method of multiplier (ADMM) [28] and preconditioned SGD [23], and its application like that in [32] is also expected. All these issues will be considered in our future work.…”

Section: Discussionmentioning

confidence: 99%

The Strength of Nesterov's Extrapolation in the Individual Convergence of Nonsmooth Optimization

Tao

Pan

et al. 2019

IEEE Trans. Neural Netw. Learning Syst.

Self Cite

View full text Add to dashboard Cite

The extrapolation strategy raised by Nesterov, which can accelerate the convergence rate of gradient descent methods by orders of magnitude when dealing with smooth convex objective, has led to tremendous success in training machine learning tasks. In this article, the convergence of individual iterates of projected subgradient (PSG) methods for nonsmooth convex optimization problems is theoretically studied based on Nesterov's extrapolation, which we name individual convergence. We prove that Nesterov's extrapolation has the strength to make the individual convergence of PSG optimal for nonsmooth problems. In light of this consideration, a direct modification of the subgradient evaluation suffices to achieve optimal individual convergence for strongly convex problems, which can be regarded as making an interesting step toward the open question about stochastic gradient descent (SGD) posed by Shamir. Furthermore, we give an extension of the derived algorithms to solve regularized learning tasks with nonsmooth losses in stochastic settings. Compared with other state-of-theart nonsmooth methods, the derived algorithms can serve as an alternative to the basic SGD especially in coping with machine learning problems, where an individual output is needed to guarantee the regularization structure while keeping an optimal rate of convergence. Typically, our method is applicable as an efficient tool for solving large-scale l 1 -regularized hinge-loss learning problems. Several comparison experiments demonstrate that our individual output not only achieves an optimal convergence rate but also guarantees better sparsity than the averaged solution.

show abstract

Section: Discussionmentioning

confidence: 99%

The Strength of Nesterov's Extrapolation in the Individual Convergence of Nonsmooth Optimization

Tao

Pan

et al. 2019

IEEE Trans. Neural Netw. Learning Syst.

Self Cite

View full text Add to dashboard Cite

show abstract

“…For comparison, the stochastic estimate of mS2GD-RBB is written in a similar way as (9). Note that for mS2GD-RBB, v k is an unbiased estimator of the gradient, i.e., from (9), we have E[v k ] = ∇P (w k ).…”

Section: B the Proposed Methodsmentioning

confidence: 99%

Accelerating Mini-batch SARAH by Step Size Rules

Yang

Chen²,

Wang

2019

Preprint

View full text Add to dashboard Cite

StochAstic Recursive grAdient algoritHm (SARAH), originally proposed for convex optimization and also proven to be effective for general nonconvex optimization, has received great attention due to its simple recursive framework for updating stochastic gradient estimates. The performance of SARAH significantly depends on the choice of step size sequence. However, SARAH and its variants often employ a best-tuned step size by mentor, which is time consuming in practice. Motivated by this gap, we proposed a variant of the Barzilai-Borwein (BB) method, referred to as the Random Barzilai-Borwein (RBB) method, to calculate step size for SARAH in the mini-batch setting, thereby leading to a new SARAH method: MB-SARAH-RBB. We prove that MB-SARAH-RBB converges linearly in expectation for strongly convex objective functions. We analyze the complexity of MB-SARAH-RBB and show that it is better than the original method. Numerical experiments on standard data sets indicate that MB-SARAH-RBB outperforms or matches state-of-the-art algorithms.

show abstract

“…Among the available methods to solve a SNEP, an elegant approach is to recast the problem as a stochastic variational inequality (SVI) [19]- [21]. The advantage of this approach is that there are many algorithms available for finding a solution of an SVI, some of them already applied to GANs [19], [22].…”

Section: B Stochastic Nash Equilibrium Problemsmentioning

confidence: 99%

Training Generative Adversarial Networks via Stochastic Nash Games

Franci

Grammatico

2023

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

Generative adversarial networks (GANs) are a class of generative models with two antagonistic neural networks: a generator and a discriminator. These two neural networks compete against each other through an adversarial process that can be modeled as a stochastic Nash equilibrium problem. Since the associated training process is challenging, it is fundamental to design reliable algorithms to compute an equilibrium. In this article, we propose a stochastic relaxed forward-backward (SRFB) algorithm for GANs, and we show convergence to an exact solution when an increasing number of data is available. We also show convergence of an averaged variant of the SRFB algorithm to a neighborhood of the solution when only a few samples are available. In both cases, convergence is guaranteed when the pseudogradient mapping of the game is monotone. This assumption is among the weakest known in the literature. Moreover, we apply our algorithm to the image generation problem.

show abstract

Stochastic Learning via Optimizing the Variational Inequalities

Cited by 12 publications

References 10 publications

The Strength of Nesterov's Extrapolation in the Individual Convergence of Nonsmooth Optimization

The Strength of Nesterov's Extrapolation in the Individual Convergence of Nonsmooth Optimization

Accelerating Mini-batch SARAH by Step Size Rules

Training Generative Adversarial Networks via Stochastic Nash Games

Contact Info

Product

Resources

About