Error bounds for approximations with deep ReLU neural networks in Ws,p norms

Abstract

We analyze approximation rates of deep ReLU neural networks for Sobolev-regular functions with respect to weaker Sobolev norms. First, we construct, based on a calculus of ReLU networks, artificial neural networks with ReLU activation functions that achieve certain approximation rates. Second, we establish lower bounds for the approximation by ReLU neural networks for classes of Sobolev-regular functions. Our results extend recent advances in the approximation theory of ReLU networks to the regime that is most relevant for applications in the numerical analysis of partial differential equations.   ,which encourages the network to encode information about the derivatives of f in its weights. The authors of [16] call this method Sobolev training and reported reduced generalization errors and better data-efficiency in a network compression task (see [31]) and in application to synthetic gradients (see [34]). In case of network compression, the approximated function f is a function realized by a possibly very large neural network N large (·|w), that has been trained for some supervised learning task and is learnt by a smaller network N small . In contrast to usual supervised learning settings, the approximated function f (·) = N large (·|w) is known and the derivatives can be computed.• Motivated by the performance of deep learning-based solutions in classical machine learning tasks and, in particular, by their ability to overcome the curse of dimension, neural networks are now also applied for the approximative solution of partial differential equations (PDEs) (see [26,36,54,59]).In [54] the authors present their deep Galerkin method for approximating solutions of high-dimensional quasilinear parabolic PDEs. For this, a functional J(f ) encoding the differential operator, boundary conditions, and initial conditions is introduced. A neural network N PDE with weights w is then trained to minimize the functional J(N PDE (w)). This is done by a discretization and randomly sampling spatial points.The theoretical foundation for approximating a function and higher-order derivatives with a neural network was already given in a less known version of the universal approximation theorem by Hornik in [32, Theorem 3]. In particular, it was shown that if the activation function ̺ is k-times continuously differentiable, non-constant, and bounded, then any k-times continuously differentiable function f and its derivatives up to order k can be uniformly approximated by a shallow neural network on compact sets. Note though that the conditions on the activation function are very restrictive and that, for example, the ReLU is not included in the above result. However, in [16], it was shown that the theorem also holds for shallow ReLU networks if k = 1. Theorem 3 in [32] was also used in [54] to show the existence of a shallow network approximating solutions of the PDEs considered in this paper. An important aspect, that is untouched by the previous approximation results is how the complexity of a network and, in particular, its depth...

Towards a regularity theory for ReLU networks – chain rule and global error estimates

“…for almost every x ∈ R d with u i (RΦ(x)) = 0. Since a finite union of nullsets is again a nullset, this proves the claim (18). The lemma follows by induction over the layers K = 1, .…”

“…In particular it allows for a stability result, i.e. Lemma III.3, the application of which will be discussed in Section V. We would also like to mention a very recent work [18] about approximation in Sobolev norms, where they deal with the issue by using a general bound for the Sobolev norm of the composition of functions from the Sobolev space W 1,∞ . Note however that this approach leads to a certain factor depending on the dimensions of the domains of the functions, which can be avoided with our method.…”

Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networks

“…For m = 1 and k = 0, Definition 6 corroborates that [87]. Accordingly, we hereby define the Lebesgue spaces of m vector-valued functions.…”

Efficient approximation of solutions of parametric linear transport equations by ReLU DNNs

“…In the definition of neural networks, we distinguish between a neural network as a set of weights and an associated function that we call the realization of the neural network. This formal approach was introduced in [48], but we recall here a slightly different formulation of [28] for neural networks that allow so-called skip connections. Let : R → R be the ReLU, i.e., = max{0, x} and let be a NN as above.…”