Probabilistic partition of unity networks: clustering based deep approximation

Abstract

Partition of unity networks (POU-Nets) have been shown capable of realizing algebraic convergence rates for regression and solution of PDEs, but require empirical tuning of training parameters. We enrich POU-Nets with a Gaussian noise model to obtain a probabilistic generalization amenable to gradient-based minimization of a maximum likelihood loss. The resulting architecture provides spatial representations of both noiseless and noisy data as Gaussian mixtures with closed form expressions for variance which provides an estimator of local error. The training process yields remarkably sharp partitions of input space based upon correlation of function values. This classification of training points is amenable to a hierarchical refinement strategy that significantly improves the localization of the regression, allowing for higher-order polynomial approximation to be utilized. The framework scales more favorably to large data sets as compared to Gaussian process regression and allows for spatially varying uncertainty, leveraging the expressive power of deep neural networks while bypassing expensive training associated with other probabilistic deep learning methods. Compared to standard deep neural networks, the framework demonstrates hp-convergence without the use of regularizers to tune the localization of partitions. We provide benchmarks quantifying performance in high/low-dimensions, demonstrating that convergence rates depend only on the latent dimension of data within high-dimensional space. Finally, we introduce a new open-source data set of PDE-based simulations of a semiconductor device and perform unsupervised extraction of a physically interpretable reduced-order basis.

Machine-learning custom-made basis functions for partial differential equations

“…However, several approaches can be used to bridge this gap, chief among which is a local spline-based interpolation method. In a somewhat related spirit, one can also make use of the recent partition of unity networks proposed in [23,37] that solve the regression problem on complex geometries by computing a partition of unity coupled with high-order polynomial expansions supported on the different partitions. Yet another approach is to make use of an extension algorithm to extend the eigenfunctions to larger, more regular domains, where Legendre expansions are available.…”

A Neural Solver for Variational Problems on Cad Geometries With Application to Electric Machine Simulation

“…An exception to the use of multiple ANNs is the method presented in Kharazmi et al (2021), which approximates the PDE with a single ANN, but making use of a separate set of test functions per subdomain. An hp-element-like approximation method based on partition of unity networks is suggested in Lee et al (2020) and Trask et al (2021); however, this method concerns a purely data-driven, supervised learning context, where ground truth values are assumed to exist for the function to be approximated, equivalently, for the solution of the PDE. Thus, the variational neural solver proposed in this paper is distinctly different than the ones presented in the aforementioned works.…”

Probabilistic partition of unity networks for high-dimensional regression problems

“…12 Moreover, although uncertainty quantification is required in many applications, it may not be naturally produced by the approximation scheme itself. 1 Instead, this work explores a general framework based on combining DNNs with polynomial approximation schemes that provides confidence regions in addition to point estimations.…”

“…2 Inspired by the deterministic model 2 and multigrid methods, 24 Trask et al showed that a natural extension of POU-Net, the probabilistic partition of unity network (PPOU-Net), can be interpreted as a mixture of experts (MoE) model, and proposed an expectation-maximization (EM) training strategy as well as a hierarchical architecture to accelerate and improve the conditioning of the training process. 1,25 Classical approximation methods enjoy the advantages of computational efficiency and convergence guarantee in solving local, low-dimensional regression problems, but they often struggle in high dimensions or as global approximants. 26 Examples of such classical methods include truncated expansions in orthogonal polynomials (e.g., Chebyshev polynomials, Legendre polynomials, Hermite polynomials) 27 and Fourier basis functions 28 , rational functions 29 , radial basis functions 30 , splines 31 , wavelets 32 , kernel methods 33 , sparse grid 34 , etc.…”

“…

We explore the probabilistic partition of unity network (PPOU-Net) model 1,2 in the context of high-dimensional regression problems. With the PPOU-Nets, the target function for any given input is approximated by a mixture of experts model, where each cluster is associated with a fixed-degree polynomial.

…”