Depth separation beyond radial functions

Abstract

High-dimensional depth separation results for neural networks show that certain functions can be efficiently approximated by two-hidden-layer networks but not by one-hidden-layer ones in high-dimensions d. Existing results of this type mainly focus on functions with an underlying radial or one-dimensional structure, which are usually not encountered in practice. The first contribution of this paper is to extend such results to a more general class of functions, namely functions with piece-wise oscillatory structure, by building on the proof strategy of (Eldan and Shamir, 2016). We complement these results by showing that, if the domain radius and the rate of oscillation of the objective function are constant, then approximation by one-hiddenlayer networks holds at a poly(d) rate for any fixed error threshold.A common theme in the proofs of depth-separation results is the fact that one-hidden-layer networks fail to approximate high-energy functions whose Fourier representation is spread in the domain. On the other hand, existing approximation results of a function by one-hidden-layer neural networks rely on the function having a sparse Fourier representation. The choice of the domain also represents a source of gaps between upper and lower approximation bounds. Focusing on a fixed approximation domain, namely the sphere S d−1 in dimension d, we provide a characterization of both functions which are efficiently approximable by one-hidden-layer networks and of functions which are provably not, in terms of their Fourier expansion.

Depth and Feature Learning are Provably Beneficial for Neural Network Discriminators

“…That is, Eldan and Shamir [2016] show that for any g is expressed as a two-layer network of width at most ce cd for some universal constant c > 0, then E x∼D (f (x) − g(x)) 2 > c. Daniely [2017] shows a simpler setting where the exponential dependency is improved to d log (d) and the non-approximation results extend to networks with polynomial weight magnitude. Safran and Shamir [2017] provide other examples where similar behavior holds, Telgarsky [2016] gives separation results beyond depth 3, and Venturi et al [2021] generalize the work of Eldan and Shamir [2016]. Note that all the results in these works concern function approximations in the L 2 (D) norm.…”

Exponential Separations in Symmetric Neural Networks

“…Many of these works center on the representational gap between twolayer and three-layer networks [3,6]. In particular, recent works have focused on generalizing the family of functions that realize these separations, to various radial functions [18] and non-radial functions [26].…”

On the Optimal Memorization Power of ReLU Neural Networks

“…In this work we show that deep networks have significantly more memorization power. Quite a few theoretical works in recent years have explored the beneficial effect of depth on increasing the expressiveness of neural networks (e.g., [23,15,33,22,12,28,38,29,10,34,6,36,35]). The benefits of depth in the context of the VC dimension is implied by, e.g., [3].…”