Untitled

Abstract

Abstract. For a common class of artificial neural networks, the mean integrated squared error between the estimated network and a target function / is shown to be bounded by where n is the number of nodes, d is the input dimension of the function, N is the number of training observations, and Cf is the first absolute moment of the Fourier magnitude distribution of /. The two contributions to this total risk are the approximation error and the estimation error. Approximation error refers to the distance between the target function and the closest neural network function of a given architecture and estimation error refers to the distance between this ideal network function and an estimated network function. With n ~ Cf(N/(dlog AT))1/2 nodes, the order of the bound on the mean integrated squared error is optimized to be O(Cf ((d/N) logN)1/2).The bound demonstrates surprisingly favorable properties of network estimation compared to traditional series and nonparametric curve estimation techniques in the case that d is moderately large. Similar bounds are obtained when the number of nodes n is not preselected as a function of Cf (which is generally not known a priori), but rather the number of nodes is optimized from the observed data by the use of a complexity regularization or minimum description length criterion. The analysis involves Fourier techniques for the approximation error, metric entropy considerations for the estimation error, and a calculation of the index of resolvability of minimum complexity estimation of the family of networks.

Developmental and evolutionary constraints on olfactory circuit selection

“…This last scaling is reasonable because when the input layer is wide enough, expansion in the hidden layer is unnecessary. In all regions, L * h shows a square-root dependence on N , as suggested from previous studies [6,8]. To further illustrate the dependence of L * h on L x and N , in Fig.…”

“…Similarly, model selection in neural networks was previously studied mostly in the large sample size limit [7,66]. The upper bound on the network size was also studied from VC (Vapnik-Chervonenkis) theory [5], and the minimum description length principle [6].…”

“…When the learning rule is unbiased, the generalization error consists of two components: the approximation error, which arises because we do not have a perfect model (we use J s rather than the true weight, J t , to model the output, y, and we may have a different nonlinearity and hidden layer size), and the estimation error, which arises because we use a finite number of training samples [6][7][8]. Inserting Eqs.…”

“…Most of that work focused on computation, in the sense that it asked what circuit, and connection strengths, lead to optimal performance on a particular task. However, the connection strengths have to be learned, and model selection theory tells us that the efficiency of learning depends crucially on architecture, especially when a limited number of trials is available [4][5][6][7][8]. In this study, we attempt to understand the organizational structure of the brain from a model selection perspective, hypothesizing that evolution optimized the brain for efficiency of learning.…”

Machine learning applied to proton radiography of high-energy-density plasmas

“…This, combined with the universality of the artificial neural network proved in [31], suggests that extreme outliers in errors can be overcome by generating more data that increases the information entropy of the data set and re-training the neural network. The relationship between errors and size of data set for various configurations of artificial neural networks is further explored in [47][48][49][50].…”