Linearized two-layers neural networks in high dimension

Abstract

We consider the problem of learning an unknown function f on the d-dimensional sphere with respect to the square loss, given i.i.d. samples {(y i , x i )} i≤n where x i is a feature vector uniformly distributed on the sphere and y i = f (x i ) + ε i . We study two popular classes of models that can be regarded as linearizations of two-layers neural networks around a random initialization: the random features model of Rahimi-Recht (RF); the neural tangent kernel model of Jacot-Gabriel-Hongler (NT). Both these approaches can also be regarded as randomized approximations of kernel ridge regression (with respect to different kernels), and enjoy universal approximation properties when the number of neurons N diverges, for a fixed dimension d.We consider two specific regimes: the approximation-limited regime, in which n = ∞ while d and N are large but finite; and the sample size-limited regime in which N = ∞ while d and n are large but finite. In the first regime, we prove that if d +δ ≤ N ≤ d +1−δ for small δ > 0, then RF effectively fits a degree-polynomial in the raw features, and NT fits a degree-( +1) polynomial. In the second regime, both RF and NT reduce to kernel methods with rotationally invariant kernels. We prove that, if the number of samples is d +δ ≤ n ≤ d +1−δ , then kernel methods can fit at most a a degree-polynomial in the raw features. This lower bound is achieved by kernel ridge regression. Optimal prediction error is achieved for vanishing ridge regularization. Contents

How rotational invariance of common kernels prevents generalization in high dimensions

“…On the one hand, Liang et al [27] prove asymptotic consistency for ground truth functions with asymptotically bounded Hilbert norms for neural tangent kernels (NTK) and inner product (IP) kernels. In contrast, Ghorbani et al [17,18] show that for uniform distributions on the product of two spheres, consistency cannot be achieved unless the ground truth is a low-degree polynomial. This polynomial approximation barrier can also be observed for random feature and neural tangent regression [17,31,32].…”

“…In contrast, Ghorbani et al [17,18] show that for uniform distributions on the product of two spheres, consistency cannot be achieved unless the ground truth is a low-degree polynomial. This polynomial approximation barrier can also be observed for random feature and neural tangent regression [17,31,32].…”

“…While recent work [2,12,19] establishes explicit asymptotic upper bounds for the bias and variance for high-dimensional linear regression, the results for kernel regression are less conclusive in the regime d/n β → c with β ∈ (0, 1). In particular, even though several papers [17,18,27] show that the variance decreases with the dimensionality of the data, the bounds on the bias are somewhat inconclusive. On the one hand, Liang et al [27] prove asymptotic consistency for ground truth functions with asymptotically bounded Hilbert norms for neural tangent kernels (NTK) and inner product (IP) kernels.…”

“…Notably, the two seemingly contradictory consistency results hold for different distributional settings and are based on vastly different proof techniques. While [27] proves consistency for general input distributions including isotropic Gaussians, the lower bounds in the papers [17,18] are limited to data that is uniformly sampled from the product of two spheres. Hence, it is a natural question to ask whether the polynomial approximation barrier is a more general phenomenon or restricted to the explicit settings studied in [17,18].…”

Forward Super-Resolution: How Can GANs Learn Hierarchical Generative Models for Real-World Distributions

“…Due to the extreme non-linearity of the networks in both the generator and the discriminator, it is highly unlikely that the training objective of GANs can be convex-concave. In particular, even if the generator and the discriminator are linear functions over prescribed feature mappings-such as the neural tangent kernel (NTK) feature mappings [3,8,9,17,18,32,35,40,41,47,51,54,65,69,92,97] -the training objective can still be non-convex-concave. 1 Even worse, unlike supervised learning where some non-convex learning problems can be shown to have no bad local minima [44], to the best of our knowledge, it still remains unclear what the qualities are of those critical points in GANs except in the most simple setting when the generator is a one-layer neural network [42,62].…”