Deep Relaxation: partial differential equations for optimizing deep neural networks

Chaudhari, Pratik; Oberman, Adam M.; Osher, Stanley; Soatto, Stefano; Carlier, Guillaume

doi:10.48550/arxiv.1704.04932

Cited by 15 publications

(32 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…3 that the best validation accuracy (72.9%) of the proposed method is higher than the one obtained with classical backpropagation (71.7%). Such a positive effect of proximal smoothing on the generalization capabilities of deep networks is consistent with the observations of Chaudhari et al (2017b). Finally, the accuracies on the test set after 50 epochs are 70.7% for ProxProp and 69.6% for BackProp which suggests that the proposed algorithm can lead to better generalization.…”

Section: Proxprop As a First-order Oraclesupporting

confidence: 85%

Proximal Backpropagation

Frerix¹,

Möllenhoff²,

Moeller³

et al. 2017

Preprint

View full text Add to dashboard Cite

We propose proximal backpropagation (ProxProp) as a novel algorithm that takes implicit instead of explicit gradient steps to update the network parameters during neural network training. Our algorithm is motivated by the step size limitation of explicit gradient descent, which poses an impediment for optimization. Prox-Prop is developed from a general point of view on the backpropagation algorithm, currently the most common technique to train neural networks via stochastic gradient descent and variants thereof. Specifically, we show that backpropagation of a prediction error is equivalent to sequential gradient descent steps on a quadratic penalty energy, which comprises the network activations as variables of the optimization. We further analyze theoretical properties of ProxProp and in particular prove that the algorithm yields a descent direction in parameter space and can therefore be combined with a wide variety of convergent algorithms. Finally, we devise an efficient numerical implementation that integrates well with popular deep learning frameworks. We conclude by demonstrating promising numerical results and show that ProxProp can be effectively combined with common first order optimizers such as Adam.

show abstract

Section: Proxprop As a First-order Oraclesupporting

confidence: 85%

Proximal Backpropagation

Frerix¹,

Möllenhoff²,

Moeller³

et al. 2017

Preprint

View full text Add to dashboard Cite

show abstract

“…The continuous-time point-of-view used in this paper gives access to general principles that govern SGD, such analyses are increasingly becoming popular (Wibisono et al, 2016;Chaudhari et al, 2017b). However, in practice, deep networks are trained for only a few epochs with discrete-time updates.…”

Section: Discussionmentioning

confidence: 99%

Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks

Chaudhari

Soatto

2018

2018 Information Theory and Applications Workshop (ITA)

Self Cite

166

237

View full text Add to dashboard Cite

Stochastic gradient descent (SGD) is widely believed to perform implicit regularization when used to train deep neural networks, but the precise manner in which this occurs has thus far been elusive. We prove that SGD minimizes an average potential over the posterior distribution of weights along with an entropic regularization term. This potential is however not the original loss function in general. So SGD does perform variational inference, but for a different loss than the one used to compute the gradients. Even more surprisingly, SGD does not even converge in the classical sense: we show that the most likely trajectories of SGD for deep networks do not behave like Brownian motion around critical points. Instead, they resemble closed loops with deterministic components. We prove that such "out-of-equilibrium" behavior is a consequence of highly non-isotropic gradient noise in SGD; the covariance matrix of mini-batch gradients for deep networks has a rank as small as 1% of its dimension. We provide extensive empirical validation of these claims, proven in the appendix.

show abstract

“…Equivalence of Entropy-SGD and Elastic-SGD. The resemblance of ( 6) and ( 7) is not a coincidence, the authors in Chaudhari et al (2017) proved that Elastic-SGD is equivalent to Entropy-SGD if the y k updates converge quickly, i.e., if the sub-objective of (6a),…”

Section: Background and Related Workmentioning

confidence: 99%

Parle: parallelizing stochastic gradient descent

Chaudhari,

Baldassi,

Zecchina

et al. 2017

Preprint

Self Cite

View full text Add to dashboard Cite

We propose a new algorithm called Parle for parallel training of deep networks that converges 2-4× faster than a data-parallel implementation of SGD, while achieving significantly improved error rates that are nearly state-of-the-art on several benchmarks including CIFAR-10 and CIFAR-100, without introducing any additional hyper-parameters. We exploit the phenomenon of flat minima that has been shown to lead to improved generalization error for deep networks. Parle requires very infrequent communication with the parameter server and instead performs more computation on each client, which makes it well-suited to both single-machine, multi-GPU settings and distributed implementations.

show abstract

Deep Relaxation: partial differential equations for optimizing deep neural networks

Cited by 15 publications

References 0 publications

Proximal Backpropagation

Proximal Backpropagation

Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks

Parle: parallelizing stochastic gradient descent

Contact Info

Product

Resources

About