Approximation theory of the MLP model in neural networks

Abstract

In this survey we discuss various approximation-theoretic problems that arise in the multilayer feedforward perceptron (MLP) model in neural networks. The MLP model is one of the more popular and practical of the many neural network models. Mathematically it is also one of the simpler models. Nonetheless the mathematics of this model is not well understood, and many of these problems are approximation-theoretic in character. Most of the research we will discuss is of very recent vintage. We will report on what has been done and on various unanswered questions. We will not be presenting practical (algorithmic) methods. We will, however, be exploring the capabilities and limitations of this model.

Universal Approximation by Ridge Computational Models and Neural Networks: A Survey

“…Relatively little is known about the approximation properties and the advantages of ridge computational models using more hidden layers. We refer the reader to [14,Section 7]. …”

“…Following [14], this can be easily explained as follows. Let us consider the particular case in which a no-hidden-layer perceptron is used for classification, i.e., when the inputs and outputs take on discrete values.…”

“…Among papers on approximation by ridge functions and consequences on neural-network approximation, see [21,15,14,22,23,24,25,26].…”

Geometric properties of the ridge function manifold

“…The study of such a manifold R of ridge functions plays a central role in both pure and applied mathematics as is manifested in the series of works [3, 6-8, 10, 12] (Temlyakov, unpublished manuscript) that concern the density of R in the space of continuous functions and the approximation of function classes by R (see also the survey of [9]). …”

On the approximation by single hidden layer feedforward neural networks with fixed weights

“…This result has shown the power of SLFNs within all possible choices of the activation function σ, provided that σ is continuous. For a detailed review of these and many other results, see [30]. In many applications, it is convenient to take the activation function σ as a sigmoidal function which is defined as lim t→−∞ σ(t) = 0 and lim t→+∞ σ(t) = 1.…”