Sharp Bounds on the Approximation Rates, Metric Entropy, and $n$-widths of Shallow Neural Networks

Near-Minimax Optimal Estimation With Shallow ReLU Neural Networks

“…We refer the reader to [9, Proposition 1] as well as the discussion after equation ( 15) in [11] for this fact. Next,…”

“…The approximation rate in Theorem 9 cannot be improved. We refer the reader to [11] for approximation lower bounds in the variation spaces of shallow neural networks. We also remark that since Theorem 9 holds in L ∞ (B d 1 ), it also holds for any L p (B d 1 ), 1 ≤ p < ∞, where the implicit constant will depend on d and p.…”

“…Invoking the result involves bounds on the L 2 (B d 1 )-metric entropy of the model class in (19). We can readily bound this metric entropy due to recent results which tightly bound the metric entropy of model classes in the variation space V 2 (B d 1 ) from [11]. The proof of Theorem 15 appears in Appendix E.…”

“…We do remark, however, that we conjecture that there is a gap between the linear minimax rate and (nonlinear) minimax rate. This follows from the fact that for the approximation problem, the gap between linear approximation methods and nonlinear approximation methods has been quantified for the variation space V 2 (Ω) (which is the same as R BV 2 (Ω) as discussed in Theorem 6) in [11]. In particular, they quantify a gap between the Kolmogorov widths (which corresponds to the approximation rate of the best linear approximation method) and the Bernstein widths (which corresponds to the approximation rate of the best nonlinear approximation method).…”

“…This provides a novel analytic characterization of this space. Using this characterization, we can apply previously derived optimal approximation rates for functions from the variation space [9], [11] to characterize the optimal approximation rates for functions in R BV 2 (Ω). The approximation rate (with respect to the…”