Translating Numerical Concepts for PDEs into Neural Architectures

Abstract

We investigate what can be learned from translating numerical algorithms into neural networks. On the numerical side, we consider explicit, accelerated explicit, and implicit schemes for a general higher order nonlinear diffusion equation in 1D, as well as linear multigrid methods. On the neural network side, we identify corresponding concepts in terms of residual networks (ResNets), recurrent networks, and U-nets. These connections guarantee Euclidean stability of specific ResNets with a transposed convolution layer structure in each block. We present three numerical justifications for skip connections: as time discretisations in explicit schemes, as extrapolation mechanisms for accelerating those methods, and as recurrent connections in fixed point solvers for implicit schemes. Last but not least, we also motivate uncommon design choices such as nonmonotone activation functions. Our findings give a numerical perspective on the success of modern neural network architectures, and they provide design criteria for stable networks.

Distributed multigrid neural solvers on megavoxel domains

“…We note that there is a marginal improvement in the loss at the same time; we show that there is a 3× improvement in training time for a very deep U-Net architecture. This ties into the theme of correlations between U-Net architecture and multigrid methods mentioned in Alt et al [1].…”

“…This approximation is plugged into the PDE, after which we invoke Galerkin's method. We multiply the PDE with a test function and reduce the differentiability requirement on 𝑢 ℎ using integration by parts: 1 ∫…”

“…Due to the fully convolutional nature of the neural network, once 𝐺 𝑛𝑛 is trained at one resolution, the forward pass of the coefficients through the network itself becomes an excellent starting point for performing interpolation (prolongation) and solving the PDE at a higher resolution. 1 For completeness, we assume 𝑢 ℎ 𝜃 ∈ 𝑉 ⊂ 𝐻 1 (𝐷) where 𝐻 1 (𝐷) denotes the Hilbert space of functions on 𝐷 that have square-integrable first derivatives. 2 Interestingly, while writing this paper, we came across work that hypothesized deep mathematical connections between numerical methods and neural nets [1], with a specific call out to a link between multigrid approaches with U-Net architectures.…”

“…1 For completeness, we assume 𝑢 ℎ 𝜃 ∈ 𝑉 ⊂ 𝐻 1 (𝐷) where 𝐻 1 (𝐷) denotes the Hilbert space of functions on 𝐷 that have square-integrable first derivatives. 2 Interestingly, while writing this paper, we came across work that hypothesized deep mathematical connections between numerical methods and neural nets [1], with a specific call out to a link between multigrid approaches with U-Net architectures. Our work anecdotally validates these assertions.…”

Distributed Multigrid Neural Solvers on Megavoxel Domains

“…We note that there is a marginal improvement in the loss at the same time; we show that there is a 3× improvement in training time for a very deep U-Net architecture. This ties into the theme of correlations between U-Net architecture and multigrid methods mentioned in Alt et al 1 . With both the architectural adaptation and Half-V cycle, in 2D spatial domain with a resolution of 512 × 512, we get a speedup of 3× over the baseline training approach at full resolution.…”