Approximations with deep neural networks in Sobolev time-space

Abstract

Solutions of evolution equation generally lies in certain Bochner-Sobolev spaces, in which the solution may has regularity and integrability properties for the time variable that can be different for the space variables. Therefore, in this paper, we develop a framework shows that deep neural networks can approximate Sobolevregular functions with respect to Bochner-Sobolev spaces. In our work we use the so-called Rectified Cubic Unit (ReCU) as an activation function in our networks, which allows us to deduce approximation results of the neural networks while avoiding issues caused by the non regularity of the most commonly used Rectivied Linear Unit (ReLU) activation function.

Integral representations of shallow neural network with Rectified Power Unit activation function

“…Let d, p ∈ N be an odd numbers, κ ∈ N 0 such that κ < p, then (p!) 1 /p R 1 (f ) = f R for any real valued function f defined on R d . In particular, R 1 (f ) is finite if and only if f is κ-order Lipschitz and f R is finite.…”

“…A theoretical understanding of which functions can be well approximated by neural networks has been studied extensively in the field of approximation theory with neural networks (e.g., [1,6,8,11,12,13,24,34,36,39,41]). In particular, integral representation techniques for shallow neural networks have received increasing attention (e.g., [7,10,15,17,19,20,23,30,32]).…”