Optimal approximation of piecewise smooth functions using deep ReLU neural networks

Abstract

We study the necessary and sufficient complexity of ReLU neural networks - in terms of depth and number of weights - which is required for approximating classifier functions in an L-sense. As a model class, we consider the set E(R) of possibly discontinuous piecewise C functions f:[-12,12]→R, where the different "smooth regions" of f are separated by C hypersurfaces. For given dimension d≥2, regularity β>0, and accuracy ε>0, we construct artificial neural networks with ReLU activation function that approximate functions from E(R) up to an L error of ε. The constructed networks have a fixed number of layers, depending only on d and β, and they have O(ε) many nonzero weights, which we prove to be optimal. For the proof of optimality, we establish a lower bound on the description complexity of the class E(R). By showing that a family of approximating neural networks gives rise to an encoder for E(R), we then prove that one cannot approximate a general function f∈E(R) using neural networks that are less complex than those produced by our construction. In addition to the optimality in terms of the number of weights, we show that in order to achieve this optimal approximation rate, one needs ReLU networks of a certain minimal depth. Precisely, for piecewise C(R) functions, this minimal depth is given - up to a multiplicative constant - by β∕d. Up to a log factor, our constructed networks match this bound. This partly explains the benefits of depth for ReLU networks by showing that deep networks are necessary to achieve efficient approximation of (piecewise) smooth functions. Finally, we analyze approximation in high-dimensional spaces where the function f to be approximated can be factorized into a smooth dimension reducing feature map τ and classifier function g - defined on a low-dimensional feature space - as f=g∘τ. We show that in this case the approximation rate depends only on the dimension of the feature space and not the input dimension.

Better Approximations of High Dimensional Smooth Functions by Deep Neural Networks with Rectified Power Units

“…Similar results for approximating functions in W k,p ([−1, 1] d ) with p < ∞ using ReLU DNNs are given by Petersen and Voigtlaender[13]. The significance of the works by Yarotsky [12] and Peterson and Voigtlaender [13] is that by using a very simple rectified nonlinearity, DNNs can obtain high order approximation property. Shallow networks do not hold such a good property.…”

“…Shallow networks do not hold such a good property. Other works show ReLU DNNs have high-order approximation property include the work by E and Wang [14] and the recent work by Opschoor et al [15], the latter one relates ReLU DNNs to high-order finite element methods.A basic fact used in the error estimate given in [12] and [13] is that x 2 , x y can be approximated by a ReLU network with O (log |ε|) layers. To remove this approximation error and the extra factor O (log |ε|) in the size of neural networks, we proposed to use rectified power units (RePU) to construct exact neural network representations of polynomials [16].…”

“…A basic fact used in the error estimate given in [12] and [13] is that x 2 , x y can be approximated by a ReLU network with O (log |ε|) layers. To remove this approximation error and the extra factor O (log |ε|) in the size of neural networks, we proposed to use rectified power units (RePU) to construct exact neural network representations of polynomials [16].…”

Towards a regularity theory for ReLU networks – chain rule and global error estimates

“…Such constructive approaches can further be found in [8], in [14] for β-cartoon-like functions, in [20] for (b, ε)holomorphic maps, and in [15] for high-frequent sinusoidal functions.…”

“…In this context it becomes relevant to not only show how well a given function of interest can be approximated by neural networks but also to extend the study to the derivative of this function. A number of recent publications [13], [14], [15] have investigated the required size of a network which is sufficient to approximate certain interesting (classes of) functions within a given accuracy. This is achieved, first, by considering the approximation of basic functions by very simple networks and, subsequently, by combining those networks in order to approximate more difficult structures.…”