Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in UQ

Abstract

We estimate the expressive power of certain deep neural networks (DNNs for short) on a class of countably-parametric, holomorphic maps [Formula: see text] on the parameter domain [Formula: see text]. Dimension-independent rates of best [Formula: see text]-term truncations of generalized polynomial chaos (gpc for short) approximations depend only on the summability exponent of the sequence of their gpc expansion coefficients. So-called [Formula: see text]-holomorphic maps [Formula: see text], with [Formula: see text] for some [Formula: see text], are known to allow gpc expansions with coefficient sequences in [Formula: see text]. Such maps arise for example as response surfaces of parametric PDEs, with applications in PDE uncertainty quantification (UQ) for many mathematical models in engineering and the sciences. Up to logarithmic terms, we establish the dimension independent approximation rate [Formula: see text] for these functions in terms of the total number [Formula: see text] of units and weights in the DNN. It follows that certain DNN architectures can overcome the curse of dimensionality when expressing possibly countably-parametric, real-valued maps with a certain degree of sparsity in the sequences of their gpc expansion coefficients. We also obtain rates of expressive power of DNNs for countably-parametric maps [Formula: see text], where [Formula: see text] is the Hilbert space [Formula: see text].

Deep Net Tree Structure for Balance of Capacity and Approximation Ability

“…and any (fixed) f * ∈ G, there exists an f ∈ H such that f * (z) = (π M f (x) − y) 2 − (f ρ (x) − y) 2 . Therefore, it follows from (8) that…”

“…Depth and structure of deep nets are two crucial factors in promoting the development of deep learning [5]. The necessity of depth has been rigorously verified from the viewpoints of approximation theory and representation theory, via showing the advantages of deep nets in localized approximation [6], sparse approximation in the frequency domain [7,8], sparse approximation in the spatial domain [9], manifold learning [10,11], hierarchical structures grasping [12,13], piecewise smoothness realization [14], universality with bounded number of parameters [15,16] and rotation invariance protection [17]. We refer the readers to Pinkus [18] and Poggio et al [19] for details on the theoretical advantages of deep nets over shallow neural networks (shallow nets).…”

Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networks

“…high-dimensional probability and statistics [90], [91] and empirical process theory [92]-core tools of ToDL, high-dimensional bounds can provide useful insights on approximation with deep ReLU FNNs fed with input data manifesting high-dimension. Toward this timely need, [93] and [94] present FNN-based approximations applicable for parametric partial differential equations. Therefore, following the lead of [93], [94] and [85]- [89], we set out to derive error bounds for a matrix-vector product approximation with deep ReLU FNNs.…”

“…Toward this timely need, [93] and [94] present FNN-based approximations applicable for parametric partial differential equations. Therefore, following the lead of [93], [94] and [85]- [89], we set out to derive error bounds for a matrix-vector product approximation with deep ReLU FNNs. This is also motivated by the fact that a matrixvector product models various research problems of wireless communications and signal processing; network science and graph signal processing; and network neuroscience and brain physics [95], [96].…”

Deep Learning of Markov Model Based Machines for Determination of Better Treatment Option Decisions for Infertile Women

“…Statistical and stochastic modeling principles were applied in deep learning algorithms to strengthen the object search capabilities or for improved model fitting in uncertainty (57,59). Boltzmann machines assist in the deep understanding of the data by linking layer level structured data and then by estimating model parameters through maximum likelihood methods (60,61).…”