A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNs

Naveh, Gadi; Ringel, Zohar

doi:10.48550/arxiv.2106.04110

Cited by 3 publications

(13 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The latter captured several strong finite DNN effects. Some empirical result on real world CNN verified the assumptions underlying this approach, namely, that the trained CNN outputs fluctuate in a nearly Gaussian manner [27].…”

Section: Introductionmentioning

confidence: 75%

“…We refer to this as kernel stability. We expect it to hold more generally, as it is essentially the statement that a latent kernel, which encompasses the feature learning effects [1,27], is stable to changing a few data points at large n.…”

Section: A Single Activated Layermentioning

confidence: 99%

“…Our aim is to obtain the function F − relating K 1 and Q 1 given f . Since feature learning can present itself through the latent kernels [23,27,1] and since typical h has many similar modes which interacting in an all-to-all manner through the non-linear termwe next adopt another type of mean-field [32] namely a Gaussian variational approximation [30]. Such approximations are common in physics (in the context of Hartree and Hatree-Fock) and statistics.…”

Section: A Single Activated Layermentioning

confidence: 99%

“…This limit is extremely hard to reach in practice, even for toy data sets. While perturbative approaches have been proposed [35,12,14,26,31], the large difference in predictions between infinite and finite DNN [18,26] is unlikely to be bridged by tractable perturbative correction [27].…”

Section: Introductionmentioning

confidence: 99%

“…[20] showed that finite width corrections could be accounted for by a re-normalization of the kernel by a scaling factor (see also [1]). Some also showed [27] that shallow linear CNNs and shallow non-linear fully-connected DNNs could be accurately analyzed using a self-consistent GP approach. The latter captured several strong finite DNN effects.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs

Seroussi¹,

Naveh²,

Ringel³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Deep neural networks (DNNs) are powerful tools for compressing and distilling information. Due to their scale and complexity, often involving billions of inter-dependent internal degrees of freedom, exact analysis approaches often fall short. A common strategy in such cases is to identify slow degrees of freedom that average out the erratic behavior of the underlying fast microscopic variables. Here, we identify such a separation of scales occurring in over-parameterized deep convolutional neural networks (CNNs) at the end of training. It implies that neuron pre-activations fluctuate in a nearly Gaussian manner with a deterministic latent kernel. While for CNNs with infinitely many channels these kernels are inert, for finite CNNs they adapt and learn from data in an analytically tractable manner. The resulting thermodynamic theory of deep learning yields accurate predictions on several deep non-linear CNN toy models. In addition, it provides new ways of analyzing and understanding CNNs.

show abstract

Section: Introductionmentioning

confidence: 75%

Section: A Single Activated Layermentioning

confidence: 99%

Section: A Single Activated Layermentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs

Seroussi¹,

Naveh²,

Ringel³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Asymptotics of representation learning in finite Bayesian neural networks

Zavatone-Veth¹,

Canatar²,

Ruben³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recent works have suggested that finite Bayesian neural networks may outperform their infinite cousins because finite networks can flexibly adapt their internal representations. However, our theoretical understanding of how the learned hidden layer representations of finite networks differ from the fixed representations of infinite networks remains incomplete. Perturbative finite-width corrections to the network prior and posterior have been studied, but the asymptotics of learned features have not been fully characterized. Here, we argue that the leading finite-width corrections to the average feature kernels for any Bayesian network with linear readout and quadratic cost have a largely universal form. We illustrate this explicitly for two classes of fully connected networks: deep linear networks and networks with a single nonlinear hidden layer. Our results begin to elucidate which features of data wide Bayesian neural networks learn to represent.

show abstract

Contrasting random and learned features in deep Bayesian linear regression

Zavatone-Veth,

Tong,

Pehlevan

2022

Preprint

View full text Add to dashboard Cite

Understanding how feature learning affects generalization is among the foremost goals of modern deep learning theory. Here, we study how the ability to learn representations affects the generalization performance of a simple class of models: deep Bayesian linear neural networks trained on unstructured Gaussian data. By comparing deep random feature models to deep networks in which all layers are trained, we provide a detailed characterization of the interplay between width, depth, data density, and prior mismatch. We show that both models display sample-wise double-descent behavior in the presence of label noise. Random feature models can also display model-wise double-descent if there are narrow bottleneck layers, while deep networks do not show these divergences. Random feature models can have particular widths that are optimal for generalization at a given data density, while making neural networks as wide or as narrow as possible is always optimal. Moreover, we show that the leading-order correction to the kernel-limit learning curve cannot distinguish between random feature models and deep networks in which all layers are trained. Taken together, our findings begin to elucidate how architectural details affect generalization performance in this simple class of deep regression models.

show abstract

A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNs

Cited by 3 publications

References 32 publications

Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs

Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs

Asymptotics of representation learning in finite Bayesian neural networks

Contrasting random and learned features in deep Bayesian linear regression

Contact Info

Product

Resources

About