Notes on ridge functions and neural networks

Abstract

To the Memory of My Parents PrefaceThese notes are about ridge functions. Recent years have witnessed a flurry of interest in these functions. Ridge functions appear in various fields and under various guises. They appear in fields as diverse as partial differential equations (where they are called plane waves), computerized tomography and statistics. These functions are also the underpinnings of many central models in neural networks.We are interested in ridge functions from the point of view of approximation theory. The basic goal in approximation theory is to approximate complicated objects by simpler objects. Among many classes of multivariate functions, linear combinations of ridge functions are a class of simpler functions. These notes study some problems of approximation of multivariate functions by linear combinations of ridge functions. We present here various properties of these functions. The questions we ask are as follows. When can a multivariate function be expressed as a linear combination of ridge functions from a certain class? When do such linear combinations represent each multivariate function? If a precise representation is not possible, can one approximate arbitrarily well? If well approximation fails, how can one compute/estimate the error of approximation, know that a best approximation exists? How can one characterize and construct best approximations? We also study properties of generalized ridge functions, which are very much related to linear superpositions and Kolmogorov's famous superposition theorem. These notes end with a few applications of ridge functions to the problem of approximation by single and two hidden layer neural networks with a restricted set of weights.We hope that these notes will be useful and interesting to both researchers and students.2 Approximation of multivariate functions by sums of univariate functions 2.1 Characterization of some bivariate function classes by formulas

Low-Rank Gradient Descent

“…There are many standard examples of ridge function losses in machine learning, e.g., least-squares regression, logistic regression, one hidden layer neural networks, etc. [15], [16]. The natural model for Principal Component Regression (PCR) is another such example [17].…”

A three layer neural network can represent any multivariate function

“…The research on this subject was carried out in two directions. In the first direction, the analysis was concentrated on approximative versions of Kolmogorov's theorem and similar results on feedforward neural networks (see, e.g., [4,9,11,16,17,18,20]). In the second direction, the precise form of the Kolmogorov superposition theorem and its relationship to neural networks were studied.…”