Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspective

Somepalli, Gowthami; Fowl, Liam; Bansal, Arpit; Yeh-Chiang, Ping; Dar, Yehuda; Baraniuk, Richard G.; Goldblum, Micah; Goldstein, Tom

doi:10.1109/cvpr52688.2022.01333

Cited by 34 publications

(44 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Implications This exciting finding supports our interpretation that faster interpolation, as promoted by overparameterization, results in model functions which are overall low-complexity, due to least (but meaningful) deviation from initialization. Our proposed interpretation corroborates the empirical observations of Gamba et al (2022a) and Somepalli et al (2022), who respectively report that large models express low curvature model functions in input space, and consistent decision boundaries. Finally, our findings extend Neyshabur et al ( 2018), who initially reported that distance from initialization decreases for overparameterized models.…”

Section: Overparameterization Constrains Complexitysupporting

confidence: 91%

On the Lipschitz Constant of Deep Networks and Double Descent

Gamba¹,

Azizpour²,

Björkman³

2023

Preprint

View full text Add to dashboard Cite

Existing bounds on the generalization error of deep networks assume some form of smooth or bounded dependence on the input variable, falling short of investigating the mechanisms controlling such factors in practice. In this work, we present an extensive experimental study of the empirical Lipschitz constant of deep networks undergoing double descent, and highlight non-monotonic trends strongly correlating with the test error. Building a connection between parameter-space and input-space gradients for SGD around a critical point, we isolate two important factors -namely loss landscape curvature and distance of parameters from initialization -respectively controlling optimization dynamics around a critical point and bounding model function complexity, even beyond the training data. Our study presents novels insights on implicit regularization via overparameterization, and effective model complexity for networks trained in practice.

show abstract

Section: Overparameterization Constrains Complexitysupporting

confidence: 91%

On the Lipschitz Constant of Deep Networks and Double Descent

Gamba¹,

Azizpour²,

Björkman³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In order to investigate how sample corruption affects margin measurements, we train several networks of increasing capacity to the point of interpolation (close to zero train error) on the widely used classification datasets MNIST [24] and CIFAR10 [25]. 'Toy problems' such as these are used extensively to probe generalization [15,23,13]. We corrupt the training data of some models using two types of noise, defined in Section 4.1, separately.…”

Section: Methodsmentioning

confidence: 99%

“…In order to find the nearest point on the decision boundary for all j, we search over each class j = i separately for each sample and choose the one with the smallest distance. As is convention for margin measurements [20,23], we use Euclidean distance * as metric, meaning the margin is given by: dist…”

Section: Formulating the Classification Marginmentioning

confidence: 99%

See 1 more Smart Citation

The Missing Margin: How Sample Corruption Affects Distance to the Boundary in ANNs

Theunissen,

Mouton,

Davel

2023

Preprint

View full text Add to dashboard Cite

Classification margins are commonly used to estimate the generalization ability of machine learning models. We present an empirical study of these margins in artificial neural networks. A global estimate of margin size is usually used in the literature. In this work, we point out seldom considered nuances regarding classification margins. Notably, we demonstrate that some types of training samples are modelled with consistently small margins while affecting generalization in different ways. By showing a link with the minimum distance to a different-target sample and the remoteness of samples from one another, we provide a plausible explanation for this observation. We support our findings with an analysis of fully-connected networks trained on noise-corrupted MNIST data, as well as convolutional networks trained on noise-corrupted CIFAR10 data.Recent studies of the relationship between margin and generalization typically measure the average margins over the training set or some sampling thereof. In this work, we ask whether the margins of individual samples tend to reflect this average behaviour. Specifically, we focus on ANN-based classification models and introduce controlled noise This work is a preprint of a published paper by the same name [1], which it subsumes. This preprint is an extended version: it contains additional empirical evidence (clean and input-corrupted CNN models

show abstract

“…In principle, two sources could cause negative flips during model upgrade (a) the stochasticity during model training, including initializations, data loading order, and optimization process (Somepalli et al, 2022). (b) the distinctions between old and new model hypotheses, including architecture and pretraining data and procedure, leading to different representation space structures and prediction behaviors in terms of decision boundaries.…”

Section: Limitations Of New Model Ensemblementioning

confidence: 99%

Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusion

Lai¹,

Mansimov²,

Xie³

et al. 2023

Preprint

View full text Add to dashboard Cite

When upgrading neural models to a newer version, new errors that were not encountered in the legacy version can be introduced, known as regression 1 errors. This inconsistent behavior during model upgrade often outweighs the benefits of accuracy gain and hinders the adoption of new models. To mitigate regression errors from model upgrade, distillation and ensemble have proven to be viable solutions without significant compromise in performance. Despite the progress, these approaches attained an incremental reduction in regression which is still far from achieving backwardcompatible model upgrade. In this work, we propose a novel method, Gated Fusion, that promotes backward compatibility via learning to mix predictions between old and new models. Empirical results on two distinct model upgrade scenarios show that our method reduces the number of regression errors by 62% on average, outperforming the strongest baseline by an average of 25%. * Work done during author's internship at AWS AI Labs. 1 Within this work, regression denotes performance degradation in software systems, instead of the statistical technique for estimating relationships among variables.

show abstract

Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspective

Cited by 34 publications

References 11 publications

On the Lipschitz Constant of Deep Networks and Double Descent

On the Lipschitz Constant of Deep Networks and Double Descent

The Missing Margin: How Sample Corruption Affects Distance to the Boundary in ANNs

Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusion

Contact Info

Product

Resources

About