Minimization of Error Functionals over Variable-Basis Functions

Abstract

Abstract. Generalized Tikhonov well-posedness is investigated for the problem of minimization of error functionals over admissible sets formed by variable-basis functions, i.e., linear combinations of a fixed number of elements chosen from a given basis without a prespecified ordering. For variablebasis functions of increasing complexity, rates of decrease of infima of error functionals are estimated. Upper bounds are derived on such rates which do not exhibit the curse of dimensionality with respect to the number of variables of admissible functions. Consequences are considered for Boolean functions and decision trees. 1. Introduction. Functionals defined as distances from (target) sets are called error functionals. Minimization of such functionals occurs in optimization tasks arising in various areas such as decision processes, system identification, machine learning, and pattern recognition.In various applications, admissible solutions over which error functionals are minimized are functions depending on a large number of variables: for example, when routing strategies have to be devised for large-scale communication and transportation networks, when an optimal closed-loop control law has to be determined for a dynamical system with high-dimensional output measurement vector and a large number of decision stages, etc. In the last decades, complex optimization problems of this kind have been approximately solved by searching suboptimal solutions over ad- Neural networks can be studied in a more general context of variable-basis functions, which also include other nonlinear families of functions such as free-node splines and trigonometric polynomials with free frequencies [17]. Families of variable-basis functions are formed by linear combinations of a fixed number of elements chosen from a given basis without a prespecified ordering [16], [17].When admissible functions depend on a large number of variables, implementation of some procedures of approximate optimization may be infeasible due to the "curse of dimensionality" [3]. For example, when optimization is performed over linear combinations of fixed-basis functions, the number of basis functions required to

Error Estimates for Approximate Optimization by the Extended Ritz Method

“…In the extended Ritz method, a nested family of linear subspaces of increasing dimensionality, which in the Ritz method approximates the set of admissible solutions, is replaced by a nested family of nonlinear approximating sets called variable-basis functions. The variable-basis approximation scheme includes a variety of nonlinear approximators such as free-nodes splines [31,Chapter 13], polynomials with free frequencies and phases [32], radial-basis-function networks with variable variances and centers [38], and feedforward neural networks [48,56].…”

“…We call n the degree of the variable-basis functions in span n G. The variable-basis approximation scheme includes free-node splines [31,Chapter 13], polynomials with free frequencies and phases [32], radial-basis-function networks with variable variances and centers [38], feedforward neural networks [48,56], and so on.…”

Optimization of approximating networks for optimal fault diagnosis

“…For recent investigations of these classes of approximators, the reader is referred to refs. [14,16,17,19,[22][23][24][25]30]. Now we apply some result from ref.…”

“…[14] as a general methodology of approximate optimization, whose theoretical properties have been investigated in refs. [l4, 16,17].…”

“…It has been so successful as to be formulated in ref. [14] as a general methodology of approximate optimization, whose theoretical properties have been investigated in refs: [14,16,17]. Approximate optimization by approximating networks allows one to reduce the solution of the optimal fault-diagnosis problem to a finite-dimensional nonlinear programming problem in which the 'free' parameters of the approximating networks have to be optimized.…”