Integral combinations of Heavisides

Abstract

Abstract:A sufficiently smooth function of d variables that decays fast enough at infinity can be represented pointwise by an integral combination of Heaviside plane waves (i.e., characteristic functions of closed half-spaces). The weight function in such a representation depends on the derivatives of the represented function. The representation is proved here by elementary techniques with separate arguments for even and odd d, and unifies and extends various results in the literature. An outline of the paper follows. Section 2 reviews neural networks and establishes notation, Section 3 discusses Green's functions and Green's second identity, and Section 4 describes functions of controlled decay and states our main theorem. We consider the 1-dimensional case in Section 5, provide necessary lemmas in Section 6, and prove the main theorem in Section 7. Section 8 has extensions and refinements of our representation and its relation to known results. The paper ends with a brief discussion. Keywords Feedforward neural networksFeedforward neural networks compute functions determined by the type of units and their interconnections. Each computational unit depends on two vector variables (an input and a parameter ), and is given by a function φ : R p × R d → R, where p and d are the dimensions of the parameter and input space respectively and R denotes the set of real numbers.One-hidden-layer networks, with hidden units based on a fixed function φ and a single linear output unit, yield functions f :where n is the number of hidden units, and w i ∈ R are the output weights and a i ∈ R p the input parameters of the i-th unit for i = 1, ..., n.A perceptron is a computational unit based on a function of the form φ ((v, b)

Integral representations of shallow neural network with Rectified Power Unit activation function

“…Since the classes of functions that can be expressed as integral with kernels are sufficiently large, many authors contributed to characterize and determine those classes, for instance, all sufficiently smooth compactly supported functions or functions decreasing sufficiently rapidly at infinity can be expressed as networks with infinitely many Heaviside perceptrons cf. [18,19,21].…”

“…A theoretical understanding of which functions can be well approximated by neural networks has been studied extensively in the field of approximation theory with neural networks (e.g., [1,6,8,11,12,13,24,34,36,39,41]). In particular, integral representation techniques for shallow neural networks have received increasing attention (e.g., [7,10,15,17,19,20,23,30,32]). Indeed, many authors have focused on the complexity control and generalization capability for shallow neural networks.…”

Dependence of Computational Models on Input Dimension: Tractability of Approximation and Optimization Tasks

“…The maximum over all with is called the Sobolev seminorm of and is denoted . The following theorem from [54] gives an integral representation of smooth functions as networks with infinitely many Heaviside perceptrons. The output weight function can be interpreted as a flow of the order through the hyperplane scaled by , which goes to zero exponentially fast with increasing.…”

“…In [53], the same formula was derived for all compactly supported functions from with odd, via an integral formula for the Dirac delta function. In [54], the representation was extended to functions of weakly-controlled decay. Representation of as a network with infinitely many perceptrons also holds for even, but the output weight function is more complicated (see [55] for the case when is in the Schwartz class and [54] for the case of satisfying certain milder conditions on smoothness and behavior at infinity).…”

Wasserstein Contraction Bounds on Closed Convex Domains with Applications to Stochastic Adaptive Control

“…[15] gives a constructive method, but only for target and activation functions in L 1 . In [16] and [17], they propose constructive methods for a class of target functions with unit step and ReLU activations respectively. In [18], functions are approximated using trigonometric polynomial ridge functions, which can then be shown in expectation to be equivalent to randomly initialized ReLU activations.…”