Generalizing Information to the Evolution of Rational Belief

Duersch, Jed A.; Catanach, Thomas A.

doi:10.3390/e22010108

Cited by 11 publications

(5 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With the same diagonal structure of P t we considered the general family of f-divergence to measure the difference between p and q. In [7], the notions of information The scaling is wrt to the standard deviation from the covariance matrix. We see some state are not converging because their error relative to the standard deviation is growing and approaches 60σ by the end of the simulation.…”

Section: Discrete-time VI Filtermentioning

confidence: 99%

Variational Kalman Filtering with Hinf-Based Correction for Robust Bayesian Learning in High Dimensions

Das¹,

Duersch²,

Catanach³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we address the problem of convergence of sequential variational inference filter (VIF) through the application of a robust variational objective and H∞-norm based correction for a linear Gaussian system. As the dimension of state or parameter space grows, performing the full Kalman update with the dense covariance matrix for a large scale system requires increased storage and computational complexity, making it impractical. The VIF approach, based on mean-field Gaussian variational inference, reduces this burden through the variational approximation to the covariance usually in the form of a diagonal covariance approximation. The challenge is to retain convergence and correct for biases introduced by the sequential VIF steps. We desire a framework that improves feasibility while still maintaining reasonable proximity to the optimal Kalman filter as data is assimilated. To accomplish this goal, a H∞norm based optimization perturbs the VIF covariance matrix to improve robustness. This yields a novel VIF-H∞ recursion that employs consecutive variational inference and H∞ based optimization steps. We explore the development of this method and investigate a numerical example to illustrate the effectiveness of the proposed filter.

show abstract

Section: Discrete-time VI Filtermentioning

confidence: 99%

Variational Kalman Filtering with Hinf-Based Correction for Robust Bayesian Learning in High Dimensions

Das¹,

Duersch²,

Catanach³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…We show how our theory of information (Duersch and Catanach, 2020), Theorem 1, allows us to derive a training objective from the information that is created when we select a prior representation, observe the training data, and either infer the posterior distribution or construct a variational approximation of it. Zhang et al (2018) provide a thorough survey of recent work on variational inference.…”

Section: Our Contributionsmentioning

confidence: 99%

“…Information-theoretic formulations of complexity can be traced back to Shannon (1948) and the concept of entropy as a measure of the uncertainty associated with sequences of discrete symbols that may be transmitted over a communication channel. In order to rigorously understand how information in our datasets relates to Bayesian inference and encoding complexity, we developed a theory of information (Duersch and Catanach, 2020) rooted in understanding information as an expectation over rational belief.…”

Section: Controlling Complexitymentioning

confidence: 99%

See 1 more Smart Citation

Parsimonious Inference

Duersch,

Catanach

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Bayesian inference provides a uniquely rigorous approach to obtain principled justification for uncertainty in predictions, yet it is difficult to articulate suitably general prior belief in the machine learning context, where computational architectures are pure abstractions subject to frequent modifications by practitioners attempting to improve results. Parsimonious inference is an information-theoretic formulation of inference over arbitrary architectures that formalizes Occam's Razor; we prefer simple and sufficient explanations. Our universal hyperprior assigns plausibility to prior descriptions, encoded as sequences of symbols, by expanding on the core relationships between program length, Kolmogorov complexity, and Solomonoff's algorithmic probability. We then cast learning as information minimization over our composite change in belief when an architecture is specified, training data are observed, and model parameters are inferred. By distinguishing model complexity from prediction information, our framework also quantifies the phenomenon of memorization.Although our theory is general, it is most critical when datasets are limited, e.g. small or skewed. We develop novel algorithms for polynomial regression and random forests that are suitable for such data, as demonstrated by our experiments. Our approaches combine efficient encodings with prudent sampling strategies to construct predictive ensembles without cross-validation, thus addressing a fundamental challenge in how to efficiently obtain predictions from data.

show abstract

“…If M m l no longer provides meaningful information at the next level, we use the next highest fidelity model in the algorithm, M m l +1 . This criteria can be formulated using a generalization of information theory [49], where the information gained about the full posterior, p (θ | D, M K ), by moving from level l to l + 1 with model M m l is:…”

Section: Information-theoretic Criteria For Model Fidelity Adaptationmentioning

confidence: 99%

Bayesian inference of Stochastic reaction networks using Multifidelity Sequential Tempered Markov Chain Monte Carlo

Catanach¹,

Vo²,

Munsky³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Stochastic reaction network models are often used to explain and predict the dynamics of gene regulation in single cells. These models usually involve several parameters, such as the kinetic rates of chemical reactions, that are not directly measurable and must be inferred from experimental data. Bayesian inference provides a rigorous probabilistic framework for identifying these parameters by finding a posterior parameter distribution that captures their uncertainty. Traditional computational methods for solving inference problems such as Markov Chain Monte Carlo methods based on classical Metropolis-Hastings algorithm involve numerous serial evaluations of the likelihood function, which in turn requires expensive forward solutions of the chemical master equation (CME). We propose an alternative approach based on a multifidelity extension of the Sequential Tempered Markov Chain Monte Carlo (ST-MCMC) sampler. This algorithm is built upon Sequential Monte Carlo and solves the Bayesian inference problem by decomposing it into a sequence of efficiently solved subproblems that gradually increase model fidelity and the influence of the observed data. We reformulate the finite state projection (FSP) algorithm, a well-known method for solving the CME, to produce a hierarchy of surrogate master equations to be used in this multifidelity scheme. To determine the appropriate fidelity, we introduce a novel information-theoretic criteria that seeks to extract the most information about the ultimate Bayesian posterior from each model in the hierarchy without inducing significant bias. This novel sampling scheme is tested with high performance computing resources using biologically relevant problems.

show abstract

Generalizing Information to the Evolution of Rational Belief

Cited by 11 publications

References 43 publications

Variational Kalman Filtering with Hinf-Based Correction for Robust Bayesian Learning in High Dimensions

Variational Kalman Filtering with Hinf-Based Correction for Robust Bayesian Learning in High Dimensions

Parsimonious Inference

Bayesian inference of Stochastic reaction networks using Multifidelity Sequential Tempered Markov Chain Monte Carlo

Contact Info

Product

Resources

About