Minimum sharpness: Scale-invariant parameter-robustness of neural networks

Ibayashi, Hikaru; Hamaguchi, Takuo; Imaizumi, Masaaki

doi:10.48550/arxiv.2106.12612

Cited by 2 publications

(2 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Related Work: Although the sharpness of the loss landscape can be calculated using several different measures (for example, the largest eigenvalue [22] or trace [23] of the Hessian of the loss), most of these measures are computationally too expensive for practical purposes. Since it has been shown that the sharpness of the loss landscape correlates with generalization [9], the main idea behind the SAM [21] algorithm is to encourage the network to seek regions where the worst loss value in a local neighbourhood is not too large (see Section 2 for more details).…”

Section: Introductionmentioning

confidence: 99%

mSAM: Micro-Batch-Averaged Sharpness-Aware Minimization

Behdin¹,

Song²,

Gupta³

et al. 2023

Preprint

View full text Add to dashboard Cite

Modern deep learning models are over-parameterized, where different optima can result in widely varying generalization performance. To account for this, Sharpness-Aware Minimization (SAM) modifies the underlying loss function to guide descent methods towards flatter minima, which arguably have better generalization abilities. In this paper, we focus on a variant of SAM known as micro-batch SAM (mSAM), which, during training, averages the updates generated by adversarial perturbations across several disjoint shards (micro batches) of a mini-batch. We extend a recently developed and well-studied general framework for flatness analysis to show that distributed gradient computation for sharpness-aware minimization theoretically achieves even flatter minima. In order to support this theoretical superiority, we provide a thorough empirical evaluation on a variety of image classification and natural language processing tasks. We also show that contrary to previous work, mSAM can be implemented in a flexible and parallelizable manner without significantly increasing computational costs. Our practical implementation of mSAM yields superior generalization performance across a wide range of tasks compared to SAM, further supporting our theoretical framework.

show abstract

Section: Introductionmentioning

confidence: 99%

mSAM: Micro-Batch-Averaged Sharpness-Aware Minimization

Behdin¹,

Song²,

Gupta³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…A flatter solution is a solution where the highest and the lowest loss values in the region do not differ too much. Sharpness measures in practice include the largest eigenvalue (Wu et al, 2018) or trace (Ibayashi et al, 2021) of the Hessian of the loss. Sharpness-Aware Minimization (SAM) (Foret et al, 2020) is an optimization framework that builds on the observation that sharpness of the training loss correlates with the generalization performance of a DNN.…”

Section: Introductionmentioning

confidence: 99%

Sharpness-Aware Minimization: An Implicit Regularization Perspective

Behdin¹,

Mazumder²

2023

Preprint

View full text Add to dashboard Cite

Sharpness-Aware Minimization (SAM) is a recent optimization framework aiming to improve the deep neural network generalization, through obtaining flatter (i.e. less sharp) solutions. As SAM has been numerically successful, recent papers have studied the theoretical aspects of the framework. In this work, we study SAM through an implicit regularization lens, and present a new theoretical explanation of why SAM generalizes well. To this end, we study the least-squares linear regression problem and show a bias-variance trade-off for SAM's error over the course of the algorithm. We show SAM has lower bias compared to Gradient Descent (GD), while having higher variance. This shows SAM can outperform GD, specially if the algorithm is stopped early, which is often the case when training large neural networks due to the prohibitive computational cost. We extend our results to kernel regression, as well as stochastic optimization and discuss how implicit regularization of SAM can improve upon vanilla training.

show abstract

Minimum sharpness: Scale-invariant parameter-robustness of neural networks

Cited by 2 publications

References 9 publications

mSAM: Micro-Batch-Averaged Sharpness-Aware Minimization

mSAM: Micro-Batch-Averaged Sharpness-Aware Minimization

Sharpness-Aware Minimization: An Implicit Regularization Perspective

Contact Info

Product

Resources

About