On the approximation of functions by tanh neural networks

Abstract

We derive bounds on the error, in high-order Sobolev norms, incurred in the approximation of Sobolev-regular as well as analytic functions by neural networks with the hyperbolic tangent activation function. These bounds provide explicit estimates on the approximation error with respect to the size of the neural networks. We show that tanh neural networks with only two hidden layers suffice to approximate functions at comparable or better rates than much deeper ReLU neural networks.

Wasserstein Generative Adversarial Uncertainty Quantification in Physics-Informed Neural Networks

“…There is still open question whether the approximation error of large scale (deep and wide) generators is small. Recently, Ryck et al [17] proved that shallow (only 1-hidden layer) but wide tanh neural networks can approximate Sobolev regular and analytic functions. His results reveal that wider neural networks have larger capacity.…”