Trajectory growth lower bounds for random sparse deep ReLU networks

Abstract

This paper considers the growth in the length of one-dimensional trajectories as they are passed through deep ReLU neural networks, which, among other things, is one measure of the expressivity of deep networks. We generalise existing results, providing an alternative, simpler method for lower bounding expected trajectory growth through random networks, for a more general class of weights distributions, including sparsely connected networks. We illustrate this approach by deriving bounds for sparse-Gaussian, sparse-uniform, and sparse-discrete-valued random nets. We prove that trajectory growth can remain exponential in depth with these new distributions, including their sparse variants, with the sparsity parameter appearing in the base of the exponent.

Deep ReLU Networks Preserve Expected Length

“…This may be done by considering a set of inputs lying along a curve and measuring the length of the corresponding curve of outputs. It has been claimed in prior literature that in a ReLU network this length distortion grows exponentially with the network's depth [24,23]. We prove that, in fact, the expected length distortion does not grow at all with depth.…”

“…Average-case analyses for ReLU networks in [24,23] showed that the expected length distortion can grow exponentially with depth, while [22] presented a similar analysis for the curvature of output trajectories. However, these results rely upon initializing the weights of the network with a distribution that also leads to exponentially large outputs [14] and exploding gradients [11], both of which are avoided by the standard "He normal" initialization [17].…”

“…Some settings of weights will lead to functions of high complexity, but these settings may be unlikely to occur in practice, depending on the distribution over weights that is under consideration. As prior work has emphasized [24,23], atypical distributions of weights can lead to exponentially high length distortion. We show that, in contrast, for deep ReLU networks trained using the standard initialization, the functions computed have low distortion in expectation and with high probability.…”