Multivariate Jackson-type inequality for a new type neural network approximation

Shao, Lin; Rong, Yuanhua; Zhao, Xu

doi:10.1016/j.apm.2014.05.018

Cited by 16 publications

(16 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where r, c 0 > 0 and x denotes the Euclidean norm of x. Denote by Lip (r,c0) the family of (r, c 0 )-Lipschitz functions satisfying (6). The Lipschitz property describes the smoothness information of f and has been adopted in vast literature [7], [28], [38], [22], [9] to quantify the approximation ability of neural networks. Denote by Lip (N,s,r,c0) the set of all f ∈ Lip (r,c0) which is s-sparse in N d partitions.…”

Section: B Sparse Approximation For Deep Netsmentioning

confidence: 99%

Generalization and Expressivity for Deep Nets

Lin

2019

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

Along with the rapid development of deep learning in practice, theoretical explanations for its success become urgent. Generalization and expressivity are two widely used measurements to quantify theoretical behaviors of deep learning. The expressivity focuses on finding functions expressible by deep nets but cannot be approximated by shallow nets with the similar number of neurons. It usually implies the large capacity. The generalization aims at deriving fast learning rate for deep nets. It usually requires small capacity to reduce the variance. Different from previous studies on deep learning, pursuing either expressivity or generalization, we take both factors into account to explore theoretical advantages of deep nets. For this purpose, we construct a deep net with two hidden layers possessing excellent expressivity in terms of localized and sparse approximation. Then, utilizing the well known covering number to measure the capacity, we find that deep nets possess excellent expressive power (measured by localized and sparse approximation) without essentially enlarging the capacity of shallow nets. As a consequence, we derive near optimal learning rates for implementing empirical risk minimization (ERM) on the constructed deep nets. These results theoretically exhibit the advantage of deep nets from learning theory viewpoints.

show abstract

Section: B Sparse Approximation For Deep Netsmentioning

confidence: 99%

Generalization and Expressivity for Deep Nets

Lin

2019

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

show abstract

“…Resonance condition (24) is fulfilled due to the definition of h n : For each g ∈ V 4n there exits at least one point z 0 ∈ X 4n such that…”

Section: Sharpness Due To Counterexamplesmentioning

confidence: 99%

“…They do not consist of linear combinations of ridge functions. A special network with four layers is introduced in [24] to obtain a Jackson estimate in terms of a first order modulus of smoothness.…”

Section: Introductionmentioning

confidence: 99%

On sharpness of error bounds for multivariate neural network approximation

Goebbels

2020

Ricerche mat

View full text Add to dashboard Cite

Single hidden layer feedforward neural networks can represent multivariate functions that are sums of ridge functions. These ridge functions are defined via an activation function and customizable weights. The paper deals with best non-linear approximation by such sums of ridge functions. Error bounds are presented in terms of moduli of smoothness. The main focus, however, is to prove that the bounds are best possible. To this end, counterexamples are constructed with a non-linear, quantitative extension of the uniform boundedness principle. They show sharpness with respect to Lipschitz classes for the logistic activation function and for certain piecewise polynomial activation functions. The paper is based on univariate results in Goebbels (Res Math 75(3):1–35, 2020. https://rdcu.be/b5mKH)

show abstract

“…From approximation theory viewpoints, parameters of FNN can be either determined via training [4] or constructed based on data directly [27]. However, the "construction" idea for FNN did not attract researchers' attention in the machine learning community, although various FNNs possessing optimal approximation property have been constructed [1], [7], [9], [10], [24], [27], [30]. The main reason is that the constructed FNN possesses superior learning capability for noisefree data only, which is usually impossible for real world applications.…”

Section: Fnn Can Be Mathematically Represented Bymentioning

confidence: 99%

“…The construction starts with selecting a set of centers (not the samples) with good geometrical distribution and generating the Voronoi partitions [15] based on them. Then, an FNN is constructed according to the well-developed constructive technique in approximation theory [8], [24], [27] by averaging outputs whose corresponding inputs are in the same partition. As the constructed FNN suffers from the well-known saturation phenomenon in the sense that the learning rate cannot be improved once the smoothness of the regression function goes beyond a specific value [19], we present a Landweber-type iterative method to overcome saturation in the last step.…”

Section: Fnn Can Be Mathematically Represented Bymentioning

confidence: 99%

Constructive Neural Network Learning

Lin

Zeng

Zhang

2019

IEEE Trans. Cybern.

Self Cite

View full text Add to dashboard Cite

In this paper, we aim at developing scalable neural network-type learning systems. Motivated by the idea of constructive neural networks in approximation theory, we focus on constructing rather than training feed-forward neural networks (FNNs) for learning, and propose a novel FNNs learning system called the constructive FNN (CFN). Theoretically, we prove that the proposed method not only overcomes the classical saturation problem for constructive FNN approximation, but also reaches the optimal learning rate when the regression function is smooth, while the state-of-the-art learning rates established for traditional FNNs are only near optimal (up to a logarithmic factor). A series of numerical simulations are provided to show the efficiency and feasibility of CFN.

show abstract

Multivariate Jackson-type inequality for a new type neural network approximation

Cited by 16 publications

References 15 publications

Generalization and Expressivity for Deep Nets

Generalization and Expressivity for Deep Nets

On sharpness of error bounds for multivariate neural network approximation

Constructive Neural Network Learning

Contact Info

Product

Resources

About