Stochastic Gradient Hamiltonian Monte Carlo for non-convex learning

Country-wide retrieval of forest structure from optical and SAR satellite imagery with deep ensembles

“…Unlike aleatoric uncertainty, it cannot simply be learned from data -instead, it generally requires marginalization over an (approximate) posterior distribution over model parameters. These methods are colloquially referred to as Bayesian deep learning (Kendall and Gal, 2017); common approaches include variational Bayesian inference (Ranganath et al, 2014;Blundell et al, 2015) and methods based on Markov chain Monte Carlo (MCMC) sampling (Neal, 1996;Welling and Teh, 2011;Chen et al, 2014). In this work, we use a deep ensemble (Lakshminarayanan et al, 2017), a method specifically developed for deep neural networks that can be understood to perform approximate Bayesian inference (Gustafsson et al, 2020).…”

A framework for variational inference and data assimilation of soil biogeochemical models using state space approximations and normalizing flows

“…Going forward, we recommend that inference approaches involving state space model approximation of stochastic SBMs be used in future biogeochemical data assimilation, fitting, and model comparison research in pursuit of superior computational stability, flexibility, and efficiency. SDE systems are far more robust than ODE systems at accommodating prior density, initial condition, and model structure proposals that are inconsistent with the true data generating process (Whitaker, 2016;Wiqvist et al, 2021) (Golightly & Kypraios, 2018), stochastic gradient Hamiltonian Monte Carlo (Chen et al, 2014), stochastic gradient langevin dynamics (Brosse et al, 2018), and stochastic gradient Markov chain Monte Carlo (Aicher et al, 2019;Nemeth & Fearnhead, 2021).…”

“…Furthermore, SDEs offer a more realistic and accurate representation of the stochasticity that is inherent to biological processes across all scales (Golightly & Wilkinson, 2011;Abs et al, 2020;Browning et al, 2020). The ability to effectively fit SDEs is an advantage of VI over many established MCMC methods; off-the-shelf MCMC implementations are frequently not designed to tolerate the noisy likelihood estimates of SDEs (Golightly & Wilkinson, 2010;Fuchs, 2013;Chen et al, 2014).…”

Efficient and generalizable tuning strategies for stochastic gradient MCMC

“…However, our proposed approach is readily generalizable to all other stochastic gradient MCMC methods, e.g. stochastic gradient Hamiltonian Monte Carlo (Chen et al 2014). Details of the general class of stochastic gradient MCMC methods presented under the complete recipe framework are given in Ma et al (2015).…”

“…We consider logistic regression on a simulated dataset with 10 dimensions and 1 million data points (details of the model and prior are in Appendix B.1). We sample from the posterior using six samplers: SGLD, SGLD with control variates (SGLD-CV, Baker et al 2019), stochastic gradient Hamiltonian Monte Carlo (SGHMC, Chen et al 2014), SGHMC-CV, stochastic gradient Nosé Hoover Thermostats (SGNHT), and SGNHT-CV (Ding et al 2014a).…”