We study the approximation properties of shallow neural networks (NN) with activation function which is a power of the rectified linear unit. Specifically, we consider the dependence of the approximation rate on the dimension and the smoothness of the underlying function to be approximated. Like the finite element method, such networks represent piecewise polynomial functions. However, we show that for sufficiently smooth functions the approximation properties of shallow ReLU k networks are much better than finite elements or wavelets, and they even overcome the curse of dimensionality more effectively than the sparse grid method. Specifically, for a sufficiently smooth function f , there exists a ReLU k-NN with n neurons which approximates f in L 2 ([0, 1] d) with O(n −(k+1) log(n)) error. Finally, we prove lower bounds showing that the approximation rates attained are optimal under the given assumptions.

We develop a theory of accelerated first-order optimization from the viewpoint of differential equations and Lyapunov functions. Building upon the work of many other researchers, we consider differential equations which model the behavior of accelerated gradient descent. Our main contribution is to provide a general framework for discretizating the differential equations to produce accelerated methods. An important novelty in our approach is the treatment of stochastic discretizations, which introduce randomness at each iteration. This leads to a unified derivation of a wide variety of methods, which include Nesterov's accelerated gradient descent, FISTA, and accelerated coordinate descent as special cases.

No abstract

No abstract

No abstract