Discussion of: “Nonparametric regression using deep neural networks with ReLU activation function”

Ghorbani, Behrooz; Mei, Song; Misiakiewicz, Theodor; Montanari, Andrea

doi:10.1214/19-aos1910

Cited by 8 publications

(5 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Further, it is shown that estimators which are not based on a composition structure do not possess the same adaptation property. For more on the results and limitations of Schmidt‐Hieber (2019), see the published discussions (Ghorbani, Mei, Misiakiewicz, and Montanari (2019), Shamir (2019), Kutyniok (2019)). Other work in this direction is Bach (2017) and Bauer and Kohler (2019).…”

Section: Deep Neural Networkmentioning

confidence: 99%

Deep Neural Networks for Estimation and Inference

Farrell

Liang

Misra

2021

ECTA

224

144

View full text Add to dashboard Cite

We study deep neural networks and their use in semiparametric inference. We establish novel nonasymptotic high probability bounds for deep feedforward neural nets. These deliver rates of convergence that are sufficiently fast (in some cases minimax optimal) to allow us to establish valid second‐step inference after first‐step estimation with deep learning, a result also new to the literature. Our nonasymptotic high probability bounds, and the subsequent semiparametric inference, treat the current standard architecture: fully connected feedforward neural networks (multilayer perceptrons), with the now‐common rectified linear unit activation function, unbounded weights, and a depth explicitly diverging with the sample size. We discuss other architectures as well, including fixed‐width, very deep networks. We establish the nonasymptotic bounds for these deep nets for a general class of nonparametric regression‐type loss functions, which includes as special cases least squares, logistic regression, and other generalized linear models. We then apply our theory to develop semiparametric inference, focusing on causal parameters for concreteness, and demonstrate the effectiveness of deep learning with an empirical application to direct mail marketing.

show abstract

Section: Deep Neural Networkmentioning

confidence: 99%

Deep Neural Networks for Estimation and Inference

Farrell

Liang

Misra

2021

ECTA

224

144

View full text Add to dashboard Cite

show abstract

“…For high-dimensional data with a large d, it is not clear when such an error bound is useful in a non-asymptotic sense. Similar concerns about this type of error bounds as established in Schmidt-Hieber (2020) are raised in the discussion by Ghorbani et al (2020), who looked at the example of additive models and pointed out that in the upper bound of the form R( fn , f 0 ) ≤ C(d)n −ǫ * log 2 n obtained in Schmidt-Hieber (2020), the d-dependence of the prefactor C(d) is not characterized. It also assumes n large enough, that is, n ≥ n 0 (d) for an unspecified n 0 (d).…”

Section: Approximation Error the Approximation Error Depends Onmentioning

confidence: 65%

“…For such an f 0 , the optimal convergence rate of the prediction error is C d n −2β/(2β+d) under mild conditions (Stone, 1982), where C d is a prefactor independent of n but depending on d and other model parameters. In low-dimensional models with a small d, the impact of C d on the convergence rate is not significant, however, in high-dimensional models with a large d, the impact of C d can be substantial, see, for example, Ghorbani et al (2020). Therefore, it is crucial to elucidate how this prefactor depends on the dimensionality so that the error bounds are meaningful in the high-dimensional settings.…”

Section: Introduction Consider a Nonparametric Regression Modelmentioning

confidence: 99%

Deep Nonparametric Regression on Approximate Manifolds: Non-Asymptotic Error Bounds with Polynomial Prefactors

Jiao

Shen

Lin

et al. 2021

Preprint

View full text Add to dashboard Cite

“…Also, the existing error bound results for the nonparametric regression estimators using deep neural networks contain prefactors that depend exponentially on the ambient dimension d of the predictor (Schmidt-Hieber, 2020;Farrell, Liang and Misra, 2021;Padilla, Tansey and Chen, 2020). This adversely affects the quality of the error bounds, especially in the high-dimensional settings when d is large (Ghorbani et al, 2020). In particular, such type of error bounds lead to a sample complexity that depends exponentially on d. Such a sample size requirement is difficult to be met even for a moderately large d.…”

Section: Introduction Consider a Nonparametric Regression Modelmentioning

confidence: 99%

Robust Nonparametric Regression with Deep Neural Networks

Shen¹,

Jiao²,

Lin³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this paper, we study the properties of robust nonparametric estimation using deep neural networks for regression models with heavy tailed error distributions. We establish the non-asymptotic error bounds for a class of robust nonparametric regression estimators using deep neural networks with ReLU activation under suitable smoothness conditions on the regression function and mild conditions on the error term. In particular, we only assume that the error distribution has a finite p-th moment with p greater than one. We also show that the deep robust regression estimators are able to circumvent the curse of dimensionality when the distribution of the predictor is supported on an approximate lower-dimensional set. An important feature of our error bound is that, for ReLU neural networks with network width and network size (number of parameters) no more than the order of the square of the dimensionality d of the predictor, our excess risk bounds depend sub-linearly on the d. Our assumption relaxes the exact manifold support assumption, which could be restrictive and unrealistic in practice. We also relax several crucial assumptions on the data distribution, the target regression function and the neural networks required in the recent literature. Our simulation studies demonstrate that the robust methods can significantly outperform the least squares method when the errors have heavy-tailed distributions and illustrate that the choice of loss function is important in the context of deep nonparametric regression.

show abstract

Discussion of: “Nonparametric regression using deep neural networks with ReLU activation function”

Cited by 8 publications

References 2 publications

Deep Neural Networks for Estimation and Inference

Deep Neural Networks for Estimation and Inference

Deep Nonparametric Regression on Approximate Manifolds: Non-Asymptotic Error Bounds with Polynomial Prefactors

Robust Nonparametric Regression with Deep Neural Networks

Contact Info

Product

Resources

About