Realizing Data Features by Deep Nets

Abstract

This paper considers the power of deep neural networks (deep nets for short) in realizing data features. Based on refined covering number estimates, we find that, to realize some complex data features, deep nets can improve the performances of shallow neural networks (shallow nets for short) without requiring additional capacity costs. This verifies the advantage of deep nets in realizing complex features. On the other hand, to realize some simple data feature like the smoothness, we prove that, up to a logarithmic factor, the approximation rate of deep nets is asymptotically identical to that of shallow nets, provided that the depth is fixed. This exhibits a limitation of deep nets in realizing simple features.

Deep Net Tree Structure for Balance of Capacity and Approximation Ability

“…Our result, exhibited in Theorem 1, establishes a covering number estimate for deep nets with arbitrarily many hidden layers and tree structures. This result improves the estimate in Guo et al [29] by reducing the exponent of R,α,L in (6), i.e.,…”

“…Using a similar approach, it was presented in Kohler and Krzyżak [34] and Lin [9] an upper bound estimate for deep nets with tree structures, five hidden layers and without the Liptchitz assumption (5) of the activation function. Recently, Kohler and Krzyzak [13] provided an estimate for covering numbers of deep nets with L-hidden layers with L ∈ N. Furthermore, covering numbers for deep nets with arbitrary structures and bounded parameters were deduced in Guo et al [29]. Our result, exhibited in Theorem 1, establishes a covering number estimate for deep nets with arbitrarily many hidden layers and tree structures.…”

“…With the restrictions imposed by (4) on deep nets, the parameters are bounded. This is indeed a necessity condition, since it can be found in Guo et al [29] and Maiorov and Pinkus [15] that there exists some h ∈ H tree 2,∞,∞ with finitely many neurons but infinite capacity (covering number).…”

“…Thus, it is a powerful capacity measurement to show the expressivity of deep nets. Secondly, covering numbers determine the limitations of approximation ability of deep nets [17,29]. Therefore, studying covering numbers of deep nets facilitates the verification of the optimality of the existing approximation results in Chui et al [6,10,17] and Mhaskar [26].…”

Deep neural networks for rotation-invariance approximation and learning

“…For these and some other RBF's, existence and uniqueness of scattered data interpolation from the linear span of {f (x−x k ) : k = 1, · · · , }, for arbitrary distinct centers {x 1 , · · · , x } and for any ∈ N, are assured. The reason for the popularity of the multiquadric RBF is fast convergence rates of the interpolants to the target function [1], and that of the Gaussian RBF is that it is commonly used as the activation function for constructing radial networks that possess the universal approximation property and other useful features (see [21], [25], [35], [39], [40], [9]) and references therein). The departure of our paper from constructing radial networks is that since RBF's are radial functions, they qualify to be target functions for our general-purpose deep nets with general activation functions.…”