Deep Network Approximation Characterized by Number of Neurons

Abstract

This paper develops simple feed-forward neural networks that achieve the universal approximation property for all continuous functions with a fixed finite number of neurons. These neural networks are simple because they are designed with a simple and computable continuous activation function σ leveraging a triangularwave function and a softsign function. We prove that σ-activated networks with width 36d(2d + 1) and depth 11 can approximate any continuous function on a d-dimensioanl hypercube within an arbitrarily small error. Hence, for supervised learning and its related regression problems, the hypothesis space generated by these networks with a size not smaller than 36d(2d + 1) × 11 is dense in the space of continuous functions. Furthermore, classification functions arising from image and signal classification are in the hypothesis space generated by σ-activated networks with width 36d(2d + 1) and depth 12, when there exist pairwise disjoint closed bounded subsets of R d such that the samples of the same class are located in the same subset.

Deep Network With Approximation Error Being Reciprocal of Width to Power of Square Root of Depth

“…Based on the proof sketch stated just above, we are ready to give the detailed proof of theorem 2 following similar ideas as in our previous work (Shen et al, 2019;Shen et al, 2020;Lu et al, 2020). The main idea of our proof is to reduce high-dimensional approximation to one-dimensional approximation via a projection.…”

A comparative investigation of neural networks in solving differential equations

“…17 In the following years, some analysis of the function approximation using single hidden layer networks have been carried out successively; see, e.g., 12,[18][19][20][21][22] However, few studies have focused on the choice of the hyperparameters of the network and how it influences the numerical solution of differential equations. Nevertheless we notice the work by Shen et al, 23 who have studied deep network approximation, characterized by the number of neurons.…”

Approximation bounds for norm constrained neural networks with applications to regression and GANs

“…The recent breakthrough of deep learning has attracted much research on the approximation theory of deep neural networks. The approximation rates of ReLU deep neural networks are well studied for many function classes, such as continuous functions (Yarotsky, 2017(Yarotsky, , 2018Shen et al, 2020), smooth functions (Yarotsky and Zhevnerchuk, 2020;Lu et al, 2021), piecewise smooth functions (Petersen and Voigtlaender, 2018), shift-invariant spaces and band-limited functions (Montanelli et al, 2019).…”

“…The recent works (Neyshabur et al, 2015;Bartlett et al, 2017;Golowich et al, 2018;Barron and Klusowski, 2019) also show that the sample complexity of deep neural networks can be controlled by certain norms of the weights. However, in the approximation theory literature, the approximation rates of deep neural networks are characterized by the number of weights (Yarotsky, 2017(Yarotsky, , 2018Yarotsky and Zhevnerchuk, 2020) or the number of neurons (Shen et al, 2020;Lu et al, 2021), rather than the size of weights.…”

“…The advantage of our approximation upper bound is that it only depends on the norm constraint so that it can be combined with the generalization bounds in (Bartlett et al, 2017;Golowich et al, 2018) and applied to over-parameterized neural networks. For comparison, in (Shen et al, 2020;Lu et al, 2021), the approximation error is bounded by the width and depth, but there is no restriction on the weights. Yarotsky (2017Yarotsky ( , 2018; Yarotsky and Zhevnerchuk (2020) obtained approximation bounds in terms of the number of non-zero weights.…”