On the Generalization Power of Overfitted Two-Layer Neural Tangent Kernel Models

Abstract

In this paper, we study the generalization performance of min ℓ2-norm overfitting solutions for the neural tangent kernel (NTK) model of a two-layer neural network. We show that, depending on the ground-truth function, the test error of overfitted NTK models exhibits characteristics that are different from the "double-descent" of other overparameterized linear models with simple Fourier or Gaussian features. Specifically, for a class of learnable functions, we provide a new upper bound of the generalization error that approaches a small limiting value, even when the number of neurons p approaches infinity. This limiting value further decreases with the number of training samples n. For functions outside of this class, we provide a lower bound on the generalization error that does not diminish to zero even when n and p are both large.

On the Generalization Power of the Overfitted Three-Layer Neural Tangent Kernel Model

“…However, these studies usually do not quantify how the generalization performance depends on the the number of neurons p. Specifically, they usually provide an upper bound on the generalization error when the number of neurons p is greater than a threshold, while the upper bound itself does not depend on p. Thus, such an upper bound cannot explain the descent behavior of NTK models. The work in Ju et al [2021] does study the decent behavior with respect to p, and is therefore the closet to our work. However, as we have explained earlier, there are crucial differences between 2 and 3 layers in both the descent behavior and the learnable set of ground-truth functions.…”

“…Based on the upper bound in Theorem 3, we have the following insights for 3-layer NTK, which are similar to those for 2-layer NTK shown in Ju et al [2021]. These similarities may reveal some intrinsic properties of the NTK models regardless of the number of layers.…”

“…In contrast, NTK models adopt features generated by non-linear activation functions (i.e., neurons of DNNs), and thus they can be viewed as an intermediate step between simple linear models and DNNs. Along this line, the work in Ju et al [2021] studies 2-layer NTK models, and shows that the 2-layer NTK model indeed exhibits better and different descent behavior in the overparameterized region, which might be closer to that of an actual neural network.…”

“…Specifically, suppose that we fix the output of the first hidden layer (i.e., h RF V,x ) and regard it as the input of a 2-layer NTK formed by the top two layers of the 3-layer neural network. By letting p 2 → ∞, we can show that the inner product between h Three V,W0,x and h Three V,W0,Xi approaches K Two ((h RF V,x ) T h RF V,Xi ) (with necessary normalization of h RF V,x and h RF V,Xi ), where K Two is exactly the kernel of 2-layer NTK in Ju et al [2021]. Second, when p 1 → ∞, we can show that (h RF V,x ) T h RF V,Xi approaches K RF (x T X i ), where K RF is exactly the kernel of the random-feature model .…”

“…Motivated by Ju et al [2021], it is of great interest to understand whether similar insights extend to deeper NTK models. In particular, in this paper we study NTK models with 3 layers.…”