Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations

Xu, Winnie; Chen, Ricky T. Q.; Li, Xuechen; Duvenaud, David

doi:10.48550/arxiv.2102.06559

Cited by 7 publications

(15 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In their case, the inference direction is the same as the generative direction, since they infer the latent SDE directly, whereas we apply Girsanov to the Feynman-Kac diffusion (opposite the generative direction). Xu et al (2021) further apply neural SDE as an infinitely deep Bayesian neural network.…”

Section: De Bruijn's Identitymentioning

confidence: 99%

A Variational Perspective on Diffusion-Based Generative Models and Score Matching

Huang¹,

Lim²,

Courville³

2021

Preprint

View full text Add to dashboard Cite

Discrete-time diffusion-based generative models and score matching methods have shown promising results in modeling high-dimensional image data. Recently, Song et al. (2021) show that diffusion processes that transform data into noise can be reversed via learning the score function, i.e. the gradient of the logdensity of the perturbed data. They propose to plug the learned score function into an inverse formula to define a generative diffusion process. Despite the empirical success, a theoretical underpinning of this procedure is still lacking. In this work, we approach the (continuous-time) generative diffusion directly and derive a variational framework for likelihood estimation, which includes continuous-time normalizing flows as a special case, and can be seen as an infinitely deep variational autoencoder. Under this framework, we show that minimizing the score-matching loss is equivalent to maximizing a lower bound of the likelihood of the plug-in reverse SDE proposed by , bridging the theoretical gap.Preprint. Under review.

show abstract

Section: De Bruijn's Identitymentioning

confidence: 99%

A Variational Perspective on Diffusion-Based Generative Models and Score Matching

Huang¹,

Lim²,

Courville³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…GoTube does not necessarily perform better in terms of average volume of the bounding balls for smaller tasks and short time horizons, as shown in Table 2. GoTube is not yet suitable for the verification of stochastic dynamical systems for instance Neural Stochastic Differential Equations (Neural SDEs) (Li et al, 2020b;Xu et al, 2021). Although GoTube is considerably more computationally efficient than existing methods, the dimensionality of the system-under-test as well as the type of numerical ODE solver exponentially affect their performance.…”

Section: Discussion Scope and Conclusionmentioning

confidence: 99%

GoTube: Scalable Stochastic Verification of Continuous-Depth Models

Gruenbacher¹,

Lechner²,

Hasani³

et al. 2021

Preprint

View full text Add to dashboard Cite

We introduce a new stochastic verification algorithm that formally quantifies the behavioural robustness of any time-continuous process formulated as a continuousdepth model. The algorithm solves a set of global optimization (Go) problems over a given time horizon to construct a tight enclosure (Tube) of the set of all process executions starting from a ball of initial states. We call our algorithm GoTube. Through its construction, GoTube ensures that the bounding tube is conservative up to a desired probability. GoTube is implemented in JAX and optimized to scale to complex continuous-depth models. Compared to advanced reachability analysis tools for time-continuous neural networks, GoTube provably does not accumulate over-approximation errors between time steps, and avoids the infamous wrapping effect inherent in symbolic techniques. We show that GoTube substantially outperforms state-of-the-art verification tools in terms of the size of the initial ball, speed, time-horizon, task completion, and scalability, on a large set of experiments. GoTube is stable and sets the state-of-the-art for its ability to scale up to time-horizons well-beyond what has been possible before.

show abstract

“…However, it should be noted that their algorithm treats the parameters of the dynamics model deterministically and thus, they can not provide the epistemic uncertainty estimates that we seek here. Note that our work is not related to the growing literature investigating SDE approximations of Bayesian Neural ODEs in the context of classification (Xu et al, 2021). Similarly to Chen et al ( 2018), these works emphasize learning a terminal state of the ODE used for other downstream tasks.…”

Section: Related Workmentioning

confidence: 94%

Distributional Gradient Matching for Learning Uncertain Neural Dynamics Models

Treven¹,

Wenk²,

Dörfler³

et al. 2021

Preprint

View full text Add to dashboard Cite

Differential equations in general and neural ODEs in particular are an essential technique in continuous-time system identification. While many deterministic learning algorithms have been designed based on numerical integration via the adjoint method, many downstream tasks such as active learning, exploration in reinforcement learning, robust control, or filtering require accurate estimates of predictive uncertainties. In this work, we propose a novel approach towards estimating epistemically uncertain neural ODEs, avoiding the numerical integration bottleneck. Instead of modeling uncertainty in the ODE parameters, we directly model uncertainties in the state space. Our algorithm -distributional gradient matching (DGM) -jointly trains a smoother and a dynamics model and matches their gradients via minimizing a Wasserstein loss. Our experiments show that, compared to traditional approximate inference methods based on numerical integration, our approach is faster to train, faster at predicting previously unseen trajectories, and in the context of neural ODEs, significantly more accurate.

show abstract

Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations

Cited by 7 publications

References 6 publications

A Variational Perspective on Diffusion-Based Generative Models and Score Matching

A Variational Perspective on Diffusion-Based Generative Models and Score Matching

GoTube: Scalable Stochastic Verification of Continuous-Depth Models

Distributional Gradient Matching for Learning Uncertain Neural Dynamics Models

Contact Info

Product

Resources

About