Fisher-Rao Metric, Geometry, and Complexity of Neural Networks

Liang, Tengyuan; Poggio, Tomaso; Rakhlin, Alexander; Stokes, James

doi:10.48550/arxiv.1711.01530

Cited by 31 publications

(43 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This line of work resulted in improved generalization bounds for deep neural networks (e.g. Dziugaite and Roy, 2016;Kawaguchi et al, 2017;Bartlett et al, 2017;Neyshabur et al, 2018Neyshabur et al, , 2017Liang et al, 2017;Arora et al, 2018;Zhou et al, 2019). This work provides further empirical evidence and alludes to more ne-grained analysis.…”

Section: Introductionmentioning

confidence: 99%

Are All Layers Created Equal?

Zhang

Bengio

Singer

2019

Preprint

View full text Add to dashboard Cite

Understanding deep neural networks has been a major research objective in recent years with notable theoretical progress. A focal point of those studies stems from the success of excessively large networks which defy the classical wisdom of uniform convergence and learnability. We study empirically the layer-wise functional structure of overparameterized deep models. We provide evidence for the heterogeneous characteristic of layers. To do so, we introduce the notion of robustness to post-training re-initialization and re-randomization. We show that the layers can be categorized as either "ambient" or "critical". Resetting the ambient layers to their initial values has no negative consequence, and in many cases they barely change throughout training. On the contrary, resetting the critical layers completely destroys the predictor and the performance drops to chanceh. Our study provides further evidence that mere parameter counting or norm accounting is too coarse in studying generalization of deep models, and atness or robustness analysis of the models needs to respect the network architectures.

show abstract

Section: Introductionmentioning

confidence: 99%

Are All Layers Created Equal?

Zhang

Bengio

Singer

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…which relates to the Fisher-Rao norm [LPRS17], a measures of model complexity. In addition, weighted 2 regularizer corresponds to anisotropic Gaussian prior on the parameters, which enjoyed empirical success in neural networks [LW17,ZTSG19].…”

Section: Related Workmentioning

confidence: 99%

On the Optimal Weighted $\ell_2$ Regularization in Overparameterized Linear Regression

2020

Preprint

View full text Add to dashboard Cite

We consider the linear model y = Xβ + with X ∈ R n×p in the overparameterized regime p > n. We estimate β via generalized (weighted) ridge regression: βλ = X X + λΣw † X y, whereΣw is the weighting matrix. Assuming a random effects model with general data covariance Σx and anisotropic prior on the true coefficients β , i.e., Eβ β = Σ β , we provide an exact characterization of the prediction risk E(y − x βλ ) 2 in the proportional asymptotic limit p/n → γ ∈ (1, ∞). Our general setup leads to a number of interesting findings. We outline precise conditions that decide the sign of the optimal setting λopt for the ridge parameter λ and confirm the implicit 2 regularization effect of overparameterization, which theoretically justifies the surprising empirical observation that λopt can be negative in the overparameterized regime. We also characterize the double descent phenomenon for principal component regression (PCR) when X and β are non-isotropic. Finally, we determine the optimal Σw for both the ridgeless (λ → 0) and optimally regularized (λ = λopt) case, and demonstrate the advantage of the weighted objective over standard ridge regression and PCR.

show abstract

“…We acknowledge that besides the algorithm-dependent approach that we follow, recent advances in learning theory aim to explain the generalization performance of neural networks from many other perspectives. Some of the most prominent ideas include bounding the network capacity by the norm of weight matrices Neyshabur et al (2015); Liang et al (2017), margin theory Bartlett et al (2017); Wei et al (2018), PAC-Bayesian theory Dziugaite and Roy (2017); ; Dziugaite and Roy (2018), network compressibility Arora et al (2018), and over-parametrization Du et al (2018); Allen-Zhu et al (2018); ; Chizat and Bach (2018). Most of these results are stated in the context of neural networks (some are tailored to networks with specific architecture), whereas our work addresses generalization in non-convex stochastic optimization in general.…”

Section: Related Workmentioning

confidence: 99%

On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning

Li¹,

Luo²,

Qiao³

2019

Preprint

View full text Add to dashboard Cite

Generalization error (also known as the out-of-sample error) measures how well the hypothesis learned from training data generalizes to previously unseen data. Proving tight generalization error bounds is a central question in statistical learning theory. In this paper, we obtain generalization error bounds for learning general non-convex objectives, which has attracted significant attention in recent years. We develop a new framework, termed Bayes-Stability, for proving algorithm-dependent generalization error bounds. The new framework combines ideas from both the PAC-Bayesian theory and the notion of algorithmic stability. Applying the Bayes-Stability method, we obtain new data-dependent generalization bounds for stochastic gradient Langevin dynamics (SGLD) and several other noisy gradient methods (e.g., with momentum, mini-batch and acceleration, Entropy-SGD). Our result recovers (and is typically tighter than) a recent result in Mou et al. (2018) and improves upon the results in Pensia et al. (2018).Our experiments demonstrate that our data-dependent bounds can distinguish randomly labelled data from normal data, which provides an explanation to the intriguing phenomena observed in Zhang et al. (2017a). We also study the setting where the total loss is the sum of a bounded loss and an additional 2 regularization term. We obtain new generalization bounds for the continuous Langevin dynamic in this setting by developing a new Log-Sobolev inequality for the parameter distribution at any time. Our new bounds are more desirable when the noisy level of the process is not small, and do not become vacuous even when T tends to infinity.

show abstract

Fisher-Rao Metric, Geometry, and Complexity of Neural Networks

Cited by 31 publications

References 5 publications

Are All Layers Created Equal?

Are All Layers Created Equal?

On the Optimal Weighted $\ell_2$ Regularization in Overparameterized Linear Regression

On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning

Contact Info

Product

Resources

About