Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis

Raginsky, Maxim; Rakhlin, Alexander; Telgarsky, Matus

doi:10.48550/arxiv.1702.03849

Cited by 48 publications

(130 citation statements)

References 26 publications

Supporting

Mentioning

123

Contrasting

Order By: Relevance

“…where ξ k is a unit Gaussian random vector in R d , η is the step size and β is an inverse temperature parameter. This update rule is known as stochastic gradient Langevin dynamics (SGLD) (Welling & Teh, 2011) and has been widely studied both in theory and practice (Raginsky et al, 2017;Zhang et al, 2017). Intuitively, (3.2) is an Euler discretization of the continuous-time diffusion equation:…”

Section: Preliminariesmentioning

confidence: 99%

“…Once F is shown to be dissipative, the machinery of (Raginsky et al, 2017;Zhang et al, 2017;Zou et al, 2020) can be adapted to show that the convergence of CS-SGLD. The majority of the remainder of the paper is devoted to proving this series of technical claims.…”

Section: Preliminariesmentioning

confidence: 99%

“…The main barrier for obtaining guarantees for recovery algorithms based on gradient descent is the non-convexity of the recovery problem induced by the generator network. Therefore, in this paper we sidestep traditional gradient descent-style optimization methods, and instead show that a very good estimate of x * can also be obtained by performing stochastic gradient Langevin Dynamics (SGLD) (Welling & Teh, 2011;Raginsky et al, 2017;Zhang et al, 2017;Zou et al, 2020). We show that this dynamics amounts to sampling from a Gibbs distribution whose energy function is precisely the reconstruction loss 1 .…”

Section: Introductionmentioning

confidence: 95%

See 2 more Smart Citations

Provable Compressed Sensing with Generative Priors via Langevin Dynamics

Nguyen¹,

Jagatap²,

Hegde³

2021

Preprint

View full text Add to dashboard Cite

Deep generative models have emerged as a powerful class of priors for signals in various inverse problems such as compressed sensing, phase retrieval and super-resolution. Here, we assume an unknown signal to lie in the range of some pre-trained generative model. A popular approach for signal recovery is via gradient descent in the low-dimensional latent space. While gradient descent has achieved good empirical performance, its theoretical behavior is not well understood. In this paper, we introduce the use of stochastic gradient Langevin dynamics (SGLD) for compressed sensing with a generative prior. Under mild assumptions on the generative model, we prove the convergence of SGLD to the true signal. We also demonstrate competitive empirical performance to standard gradient descent.

show abstract

Section: Preliminariesmentioning

confidence: 99%

Section: Preliminariesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 95%

See 1 more Smart Citation

Provable Compressed Sensing with Generative Priors via Langevin Dynamics

Nguyen¹,

Jagatap²,

Hegde³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…To solve (10), we apply a stochastic gradient descent (SGD) algorithm. In general, SGD finds a local minimum of a nonconvex objective function (Ge et al, 2015) but a global minimizer in some special situations (Raginsky et al, 2017;.…”

Section: Theoretical Examplementioning

confidence: 99%

Significance tests of feature relevance for a black-box learner

Dai,

Shen,

Pan

2021

Preprint

View full text Add to dashboard Cite

An exciting recent development is the uptake of deep learning in many scientific fields, where the objective is seeking novel scientific insights and discoveries. To interpret a learning outcome, researchers perform hypothesis testing for explainable features to advance scientific domain knowledge. In such a situation, testing for a blackbox learner poses a severe challenge because of intractable models, unknown limiting distributions of parameter estimates, and high computational constraints. In this article, we derive two consistent tests for the feature relevance of a blackbox learner. The first one evaluates a loss difference with perturbation on an inference sample, which is independent of an estimation sample used for parameter estimation in model fitting. The second further splits the inference sample into two but does not require data perturbation. Also, we develop their combined versions by aggregating the order statistics of the p-values based on repeated sample splitting. To estimate the splitting ratio and the perturbation size, we develop adaptive splitting schemes for suitably controlling the Type I error subject to computational constraints. By deflating the bias-sd-ratio, we establish asymptotic null distributions of the test statistics and their consistency in terms of statistical power. Our theoretical power analysis and simulations indicate that the one-split test is more powerful than the two-split test, though the latter is easier to apply for large datasets. Moreover, the combined tests are more stable while compensating for a power loss by repeated sample splitting. Numerically, we demonstrate the utility of the proposed tests on two benchmark examples. Accompanying this paper is our Python library dnn-inference (https://dnn-inference.readthedocs. io/en/latest/) that implements the proposed tests.

show abstract

“…There is a rich field in the literature on discretizing Langevin diffusion [Dalalyan, 2016, Raginsky et al, 2017, Dalalyan, 2017 primarily used for designing efficient sampling algorithms -a large fraction of them analyzing the above discretization. We will mostly rely on tools from Bubeck et al [2018].…”

Section: D1 Bounding the Error In The Euler-mariyama Discretizationmentioning

confidence: 99%

The Effects of Invertibility on the Representational Complexity of Encoders in Variational Autoencoders

Pareek,

Risteski

2021

Preprint

View full text Add to dashboard Cite

Training and using modern neural-network based latent-variable generative models (like Variational Autoencoders) often require simultaneously training a generative direction along with an inferential (encoding) direction, which approximates the posterior distribution over the latent variables. Thus, the question arises: how complex does the inferential model need to be, in order to be able to accurately model the posterior distribution of a given generative model?In this paper, we identify an important property of the generative map impacting the required size of the encoder. We show that if the generative map is "strongly invertible" (in a sense we suitably formalize), the inferential model need not be much more complex. Conversely, we prove that there exist non-invertible generative maps, for which the encoding direction needs to be exponentially larger (under standard assumptions in computational complexity). Importantly, we do not require the generative model to be layerwise invertible, which a lot of the related literature assumes and isn't satisfied by many architectures used in practice (e.g. convolution and pooling based networks). Thus, we provide theoretical support for the empirical wisdom that learning deep generative models is harder when data lies on a low-dimensional manifold.

show abstract

Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis

Cited by 48 publications

References 26 publications

Provable Compressed Sensing with Generative Priors via Langevin Dynamics

Provable Compressed Sensing with Generative Priors via Langevin Dynamics

Significance tests of feature relevance for a black-box learner

The Effects of Invertibility on the Representational Complexity of Encoders in Variational Autoencoders

Contact Info

Product

Resources

About