Neural Networks with Small Weights and Depth-Separation Barriers

Abstract

In studying the expressiveness of neural networks, an important question is whether there are functions which can only be approximated by sufficiently deep networks, assuming their size is bounded. However, for constant depths, existing results are limited to depths 2 and 3, and achieving results for higher depths has been an important open question. In this paper, we focus on feedforward ReLU networks, and prove fundamental barriers to proving such results beyond depth 4, by reduction to open problems and natural-proof barriers in circuit complexity. To show this, we study a seemingly unrelated problem of independent interest: Namely, whether there are polynomially-bounded functions which require super-polynomial weights in order to approximate with constant-depth neural networks. We provide a negative and constructive answer to that question, by showing that if a function can be approximated by a polynomially-sized, constant depth k network with arbitrarily large weights, it can also be approximated by a polynomially-sized, depth 3k + 3 network, whose weights are polynomially bounded.

On the Optimal Memorization Power of ReLU Neural Networks

“…In this work we show that deep networks have significantly more memorization power. Quite a few theoretical works in recent years have explored the beneficial effect of depth on increasing the expressiveness of neural networks (e.g., [23,15,33,22,12,28,38,29,10,34,6,36,35]). The benefits of depth in the context of the VC dimension is implied by, e.g., [3].…”

Size and Depth Separation in Approximating Benign Functions with Neural Networks

“…Overwhelming empirical evidence indicates that deeper networks tend to perform better than shallow ones. Quite a few theoretical works in recent years have explored the beneficial effect of depth on increasing the expressiveness of neural networks (e.g., Martens et al [2013], Eldan and Shamir [2016], Telgarsky [2016], Liang and Srikant [2016], Daniely [2017], Safran and Shamir [2017], Yarotsky [2017], Safran et al [2019], Vardi and Shamir [2020], Bresler and Nagaraj [2020]). A main focus is on depth separation, namely, showing that there is a function f : R d → R that can be approximated by a poly(d)-sized network of a given depth, with respect to some input distribution, but cannot be approximated by poly(d)-sized networks of a smaller depth.…”

“…Depth separation between depth 2 and 3 is known Shamir, 2016, Daniely, 2017] 1 already for natural functions. A complexity-theoretic barrier to proving separation between two constant depths beyond depth 4 was established in Vardi and Shamir [2020]. A construction shown by Telgarsky [2016] gives separation between poly(d)-sized networks of a constant depth, and poly(d)-sized networks of some nonconstant depth.…”

“…Moreover, if we wish to prove the result for depth k ≥ 6, it would require overcoming a natural-proofs barrier, a concept from computational complexity which indicates that such a proof would be very hard to find. We note that Vardi and Shamir [2020] gave a barrier to proving depth separation, and we give a barrier already to proving depth lower bounds. Thus, while the barrier of Vardi and Shamir [2020] holds only for separation between two constant depths, our barrier applies also to separation between constant and nonconstant depths.…”

“…We note that Vardi and Shamir [2020] gave a barrier to proving depth separation, and we give a barrier already to proving depth lower bounds. Thus, while the barrier of Vardi and Shamir [2020] holds only for separation between two constant depths, our barrier applies also to separation between constant and nonconstant depths.…”