Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysis

Tsuzuku, Yusuke; Sato, Issei; Sugiyama, Masashi

doi:10.48550/arxiv.1901.04653

Cited by 9 publications

(10 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Related Workmentioning

confidence: 99%

“…The Hessians lack of reparameterisation invariance [7], has subjected its use for predicting generalisation to criticism [29,38,32]. Normalized definitions of flatness, have been introduced Tsuzuku et al [38], Rangamani et al [32] in a PAC-Bayesian framework, although Rangamani et al [32] note that empirically Hessian based sharpness measures correlate with generalisation. Smith and Le [37] argue that although sharpness can be manipulated, the Bayesian log evidence which they approximate as the log determinant of the Hessian i log(λ i /γ) is invariant to such reparameterisation, where γ is the L2 regularisation co-efficient.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Flatness is a False Friend

Granziol

2020

Preprint

View full text Add to dashboard Cite

Hessian based measures of flatness, such as the trace, Frobenius and spectral norms, have been argued, used and shown to relate to generalisation. In this paper we demonstrate that for feed forward neural networks under the cross entropy loss, we would expect low loss solutions with large weights to have small Hessian based measures of flatness. This implies that solutions obtained using L2 regularisation should in principle be sharper than those without, despite generalising better. We show this to be true for logistic regression, multi-layer perceptrons, simple convolutional, pre-activated and wide residual networks on the MNIST and CIFAR-100 datasets. Furthermore, we show that for adaptive optimisation algorithms using iterate averaging, on the VGG-16 network and CIFAR-100 dataset, achieve superior generalisation to SGD but are 30× sharper. This theoretical finding, along with experimental results, raises serious questions about the validity of Hessian based sharpness measures in the discussion of generalisation. We further show that the Hessian rank can be bounded by the a constant times number of neurons multiplied by the number of classes, which in practice is often a small fraction of the network parameters. This explains the curious observation that many Hessian eigenvalues are either zero or very near zero which has been reported in the literature.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Flatness is a False Friend

Granziol

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Some researchers specifically study flatness itself. They try to measure flatness (Hochreiter and Schmidhuber, 1997;Keskar et al, 2017;Sagun et al, 2017;Yao et al, 2018), rescale flatness (Tsuzuku et al, 2019), and find flatter minima (Hoffer et al, 2017;Chaudhari et al, 2017;He et al, 2019b). But we still lack a quantitative theory that answers why deep learning find flat minima with such a high probability.…”

Section: Introductionmentioning

confidence: 99%

A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima

Xie,

Sato,

Sugiyama

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Stochastic Gradient Descent (SGD) and its variants are mainstream methods for training deep networks in practice. SGD is known to find flat minima with a large neighboring region in parameter space from which each weight vector has similar small error. However, the quantitative theory behind stochastic gradients still remains to be further investigated. In this paper, we focus on a fundamental problem in deep learning, "How can deep learning select flat minima among so many minima?" To answer the question quantitatively, we develop a density diffusion theory (DDT) to reveal how minima selection quantitatively depends on minima sharpness, gradient noise and hyperparameters. One of the most interesting findings is that stochastic gradient noise from SGD can accelerate escaping sharp minima exponentially in terms of top eigenvalues of minima hessians, while white noise can only accelerate escaping sharp minima polynomially in terms of the determinants of minima hessians. We also find large-batch training requires exponentially many iterations to escape sharp minima in terms of batch size. We present direct empirical evidence supporting the proposed theoretical results.Preprint. Under review.

show abstract

“…The most prominent critique of the sharp-minima hypothesis comes from Dinh et al [2017], who proves that one can increase the sharpness of any given minima by reparametrizing the network. Similar criticism can be found in theoretical PAC-Bayes work [Tsuzuku et al, 2019, Rangamani et al, 2019, Yi et al, 2019, Neyshabur et al, 2017, that only provides experiments for large-vssmall batch sizes where standard sharpness metrics work well in practice. Van Laarhoven [2017] noted how WD would increase the relative size of gradient updates.…”

Section: Discussionmentioning

confidence: 61%

Understanding Decoupled and Early Weight Decay

Björck¹,

Weinberger²,

Gomes³

2020

Preprint

View full text Add to dashboard Cite

Weight decay (WD) is a traditional regularization technique in deep learning, but despite its ubiquity, its behavior is still an area of active research. Golatkar et al. have recently shown that WD only matters at the start of the training in computer vision, upending traditional wisdom. Loshchilov et al. show that for adaptive optimizers, manually decaying weights can outperform adding an l 2 penalty to the loss. This technique has become increasingly popular and is referred to as decoupled WD. The goal of this paper is to investigate these two recent empirical observations. We demonstrate that by applying WD only at the start, the network norm stays small throughout training. This has a regularizing effect as the effective gradient updates become larger. However, traditional generalizations metrics fail to capture this effect of WD, and we show how a simple scale-invariant metric can. We also show how the growth of network weights is heavily influenced by the dataset and its generalization properties. For decoupled WD, we perform experiments in NLP and RL where adaptive optimizers are the norm. We demonstrate that the primary issue that decoupled WD alleviates is the mixing of gradients from the objective function and the l 2 penalty in the buffers of Adam (which stores the estimates of the first-order moment). Adaptivity itself is not problematic and decoupled WD ensures that the gradients from the l 2 term cannot "drown out" the true objective, facilitating easier hyperparameter tuning.

show abstract

Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysis

Cited by 9 publications

References 10 publications

Flatness is a False Friend

Flatness is a False Friend

A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima

Understanding Decoupled and Early Weight Decay

Contact Info

Product

Resources

About