Constructive Deep ReLU Neural Network Approximation

On the approximation of functions by tanh neural networks

“…Moreover, we also consider the case of analytic functions and will prove that a two hidden layer tanh neural network suffices to approximate an analytic function at an exponential rate, in terms of the network width, even in Sobolev norms. This result provides an improvement over available results for the approximation of analytic functions by ReLU neural networks [65,55,22] and also neural networks with smooth activation functions [44] and further illustrate the powers of rather shallow tanh networks at approximating smooth functions. Finally, we also derive explicit bounds on the width of the tanh neural networks as well as asymptotic bounds on their weights, thus paving the way for bounds on the generalization error for these neural networks.…”

“…Exponential convergence (in terms of network size) of neural networks for analytic functions in the L ∞ -norm was first proven in [44] for neural networks with smooth activation functions and in [65] for ReLU neural networks. In [55,22], the authors prove exponential convergence in W 1,∞ -norm for ReLU neural networks. We compare our results for approximation of analytic functions with these papers in Table 2.…”

“…A seminal work in the direction is [66], where the author derived explicit estimates on the size (width and depth) of a neural network with a ReLU activation function for approximating Lipschitz functions to any given accuracy in the L ∞ -norm. Expressivity results for such ReLU neural networks in Sobolev norms were presented in [19,22,55] and references therein, see also [36,50,54,61,67] and references therein for further approximation results for ReLU and related ReQU and RePU activation functions.…”

Deep ReLU neural network approximation in Bochner spaces and applications to parametric PDEs

“…In the last several years, there has been a number of interesting papers that addressed the role of depth and architecture of deep neural networks in approximating functions that possess special regularity properties such as analytic functions [20,38], differentiable functions [45,52], oscillatory functions [29], functions in Sobolev or Besov spaces [1,27,30,53]. High-dimensional approximations by deep neural networks have been studied in [39,48,16,17], and their applications to high-dimensional PDEs in [47,21,43,31,25,26,28]. Most of these papers used deep ReLU (Rectified Linear Unit) neural networks since the rectified linear unit is a simple and preferable activation function in many applications.…”

“…Recently, a number of works have been devoted to various problems and methods of deep neural network approximation for parametric and stochastic PDEs such as dimensionality reduction [51], deep neural network expression rates for the Taylor generalized polynomial chaos expansion (gpc) of solutions to parametric elliptic PDEs [46], reduced basis methods [36] the problem of learning the discretized parameter-to-solution map in practice [24], Bayesian PDE inversion [42,32,31], etc. In particular, in [46] the authors proved dimension-independent deep neural network expression rate bounds of the uniform approximation of solution to parametric elliptic PDE with affine inputs on I ∞ := [−1, 1] ∞ based on n-term truncations of the non-orthogonal Taylor gpc expansion.…”